Re: saving record into different tables?

[l vic]

Ok, I've got "split" working but that still leaves me with the dilemma: i
have to write multiple rows for "mylist" into the same table... Note that I
am working with db (Phoenix Query Server)  that doesn't support "batching"
of multiple records in one insert as mysql. Any advise?


On Wed, Sep 5, 2018 at 10:04 PM Matt Burgess  wrote:

> V,
>
> Perhaps ironically but for the greater good, NiFi (IMHO, with record
> processors) performs better when you don't touch the content but just
> describe what you want from it. So in your case you could send the
> content to two different branches, one could be a PutDatabaseRecord
> with a reader schema that ignores the "mylist" element, and the other
> could use Jolt / SplitRecord / ForkRecord to "hoist" the fields from
> the "mylist" element into a root array/RecordSet, then use
> PutDatabaseRecord to process it. Then you don't need a "split" or "get
> into attribute" or any other operation that isn't in line with your
> business logic.
>
> Regards,
> Matt
>
> On Fri, Aug 31, 2018 at 7:34 PM l vic  wrote:
> >
> > Hi,
> > I have a json record that contains array "mylst":
> > {
> > "id": 0,
> > "name": "Root",
> >  "mylist": [{
> >   "id": 10,
> >   "info": "2am-3am"
> > },
> > {
> > "id": 11,
> > "info": "3AM-4AM"
> > },
> > {
> > "id": 12,
> > "info": "4am-5am"
> > }]
> >
> > }
> > I have to save root data into one db table and array into another... Can
> someone recommend an approach to "splitting" of record for 2 different
> database writers?
> > Thank you,
> > V
>

Re: Nifi Publish/Consumer Kafka and Azure Event Hub

[Bryan Bende]

Hello,

Any user defined properties in the processor should be passed along to
Kafka so you should be able to add a new property with the name
sasl.mechanism and the value PLAIN, without that the processors will assume
GSSAPI.

On Wed, Sep 5, 2018 at 9:55 PM João Henrique Freitas 
wrote:

>
> Hi James,
>
> I did the same sequence of steps that you wrote.
>
> When the kafka processor starts I see this:
>
> 2018-09-05 22:44:02,999 INFO [Timer-Driven Process Thread-1]
> o.a.k.clients.producer.ProducerConfig ProducerConfig values:
> acks = 0
> batch.size = 16384
> bootstrap.servers = [xyz.servicebus.windows.net:9093]
> buffer.memory = 33554432
> client.id =
> compression.type = none
> connections.max.idle.ms = 54
> enable.idempotence = false
> interceptor.classes = null
> key.serializer = class
> org.apache.kafka.common.serialization.ByteArraySerializer
> linger.ms = 0
> max.block.ms = 5000
> max.in.flight.requests.per.connection = 5
> max.request.size = 1048576
> metadata.max.age.ms = 30
> metric.reporters = []
> metrics.num.samples = 2
> metrics.recording.level = INFO
> metrics.sample.window.ms = 3
> partitioner.class = class
> org.apache.kafka.clients.producer.internals.DefaultPartitioner
> receive.buffer.bytes = 32768
> reconnect.backoff.max.ms = 1000
> reconnect.backoff.ms = 50
> request.timeout.ms = 3
> retries = 0
> retry.backoff.ms = 100
> sasl.jaas.config = [hidden]
> sasl.kerberos.kinit.cmd = /usr/bin/kinit
> sasl.kerberos.min.time.before.relogin = 6
> sasl.kerberos.service.name = kafka
> sasl.kerberos.ticket.renew.jitter = 0.05
> sasl.kerberos.ticket.renew.window.factor = 0.8
> sasl.mechanism = GSSAPI
> security.protocol = SASL_SSL
> send.buffer.bytes = 131072
> ssl.cipher.suites = null
> ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
> ssl.endpoint.identification.algorithm = null
> ssl.key.password = [hidden]
> ssl.keymanager.algorithm = SunX509
> ssl.keystore.location = /home/joaohf/bin/nifi-1.7.1/cacerts
> ssl.keystore.password = [hidden]
> ssl.keystore.type = JKS
> ssl.protocol = TLS
> ssl.provider = null
> ssl.secure.random.implementation = null
> ssl.trustmanager.algorithm = PKIX
> ssl.truststore.location = null
> ssl.truststore.password = null
> ssl.truststore.type = JKS
> transaction.timeout.ms = 6
> transactional.id = null
> value.serializer = class
> org.apache.kafka.common.serialization.ByteArraySerializer
>
>
> The Azure documentation says:
>
> security.protocol=SASL_SSL
> sasl.mechanism=PLAIN
>
> But Nifi shows:
>
> security.protocol = SASL_SSL
> sasl.mechanism = GSSAPI
>
>
> I'm using nifi 1.7.1.
>
> How I could set sasl.mechanism to PLAIN and use SASL_SSL ? Maybe a specific 
> combination that Kafka processor hasn't support yet?
>
>
> Thanks.
>
>
>
>
>
>
>
> On Wed, Sep 5, 2018 at 5:59 PM James Srinivasan <
> james.sriniva...@gmail.com> wrote:
>
>> I've not tried this myself, but once you have a working JAAS config
>> (from
>> https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-quickstart-kafka-enabled-event-hubs#send-and-receive-messages-with-kafka-in-event-hubs
>> ),
>> set the corresponding protocol and mechanism properties in the NiFi
>> processor, and put the content of sasl.jaas.config in a file and
>> reference it from NiFi's bootstrap.conf as indicated by the second
>> link you found.
>>
>> Good luck!
>> On Wed, 5 Sep 2018 at 15:54, João Henrique Freitas 
>> wrote:
>> >
>> >
>> > Hello!
>> >
>> > I'm exploring Azure Event Hub with Kafka support. I know that's in
>> preview.
>> >
>> > But I would like to know how to use PublishKafka with this
>> configuration:
>> >
>> >
>> https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-quickstart-kafka-enabled-event-hubs
>> >
>> > I don't know how to configure kafka processor with authentication
>> parameters like:
>> >
>> > security.protocol=SASL_SSL
>> > sasl.mechanism=PLAIN
>> >
>> sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule
>> required username="$ConnectionString"
>> password="{YOUR.EVENTHUBS.CONNECTION.STRING}";
>> >
>> >
>> > Should I follow this?
>> >
>> >
>> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-kafka-1-0-nar/1.7.1/org.apache.nifi.processors.kafka.pubsub.PublishKafka_1_0/additionalDetails.html
>> >
>> > Or maybe its necessary do a patch in Publish/Consumer KafkaProcessor  ?
>> >
>> > Best regards,
>> >
>> > --
>> > João Henrique Ferreira de Freitas - joaohf_at_gmail.com
>> > Campinas-SP-Brasil
>>
>
>
> --
> João Henrique Ferreira de Freitas - joaohf_at_gmail.com
> Campinas-SP-Brasil
>
-- 
Sent from Gmail Mobile

Re: Nifi Publish/Consumer Kafka and Azure Event Hub

[João Henrique Freitas]

Hi James,

I did the same sequence of steps that you wrote.

When the kafka processor starts I see this:

2018-09-05 22:44:02,999 INFO [Timer-Driven Process Thread-1]
o.a.k.clients.producer.ProducerConfig ProducerConfig values:
acks = 0
batch.size = 16384
bootstrap.servers = [xyz.servicebus.windows.net:9093]
buffer.memory = 33554432
client.id =
compression.type = none
connections.max.idle.ms = 54
enable.idempotence = false
interceptor.classes = null
key.serializer = class
org.apache.kafka.common.serialization.ByteArraySerializer
linger.ms = 0
max.block.ms = 5000
max.in.flight.requests.per.connection = 5
max.request.size = 1048576
metadata.max.age.ms = 30
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 3
partitioner.class = class
org.apache.kafka.clients.producer.internals.DefaultPartitioner
receive.buffer.bytes = 32768
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 3
retries = 0
retry.backoff.ms = 100
sasl.jaas.config = [hidden]
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 6
sasl.kerberos.service.name = kafka
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.mechanism = GSSAPI
security.protocol = SASL_SSL
send.buffer.bytes = 131072
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
ssl.endpoint.identification.algorithm = null
ssl.key.password = [hidden]
ssl.keymanager.algorithm = SunX509
ssl.keystore.location = /home/joaohf/bin/nifi-1.7.1/cacerts
ssl.keystore.password = [hidden]
ssl.keystore.type = JKS
ssl.protocol = TLS
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
transaction.timeout.ms = 6
transactional.id = null
value.serializer = class
org.apache.kafka.common.serialization.ByteArraySerializer


The Azure documentation says:

security.protocol=SASL_SSL
sasl.mechanism=PLAIN

But Nifi shows:

security.protocol = SASL_SSL
sasl.mechanism = GSSAPI


I'm using nifi 1.7.1.

How I could set sasl.mechanism to PLAIN and use SASL_SSL ? Maybe a
specific combination that Kafka processor hasn't support yet?


Thanks.







On Wed, Sep 5, 2018 at 5:59 PM James Srinivasan 
wrote:

> I've not tried this myself, but once you have a working JAAS config
> (from
> https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-quickstart-kafka-enabled-event-hubs#send-and-receive-messages-with-kafka-in-event-hubs
> ),
> set the corresponding protocol and mechanism properties in the NiFi
> processor, and put the content of sasl.jaas.config in a file and
> reference it from NiFi's bootstrap.conf as indicated by the second
> link you found.
>
> Good luck!
> On Wed, 5 Sep 2018 at 15:54, João Henrique Freitas 
> wrote:
> >
> >
> > Hello!
> >
> > I'm exploring Azure Event Hub with Kafka support. I know that's in
> preview.
> >
> > But I would like to know how to use PublishKafka with this configuration:
> >
> >
> https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-quickstart-kafka-enabled-event-hubs
> >
> > I don't know how to configure kafka processor with authentication
> parameters like:
> >
> > security.protocol=SASL_SSL
> > sasl.mechanism=PLAIN
> > sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule
> required username="$ConnectionString"
> password="{YOUR.EVENTHUBS.CONNECTION.STRING}";
> >
> >
> > Should I follow this?
> >
> >
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-kafka-1-0-nar/1.7.1/org.apache.nifi.processors.kafka.pubsub.PublishKafka_1_0/additionalDetails.html
> >
> > Or maybe its necessary do a patch in Publish/Consumer KafkaProcessor  ?
> >
> > Best regards,
> >
> > --
> > João Henrique Ferreira de Freitas - joaohf_at_gmail.com
> > Campinas-SP-Brasil
>


-- 
João Henrique Ferreira de Freitas - joaohf_at_gmail.com
Campinas-SP-Brasil

Re: Anyone using HashAttribute?

[Andy LoPresto]

Thanks Dan. The blogs encourage us to keep building.

Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Sep 5, 2018, at 4:25 PM, dan young  wrote:
> 
> Heya Andy,
> 
> yes, that seems legit...we'll make it work on our side...
> 
> Keep up the awesome work on NiFi, powers all of our ETL here now :)
> 
> Dano
> 
> On Wed, Sep 5, 2018 at 5:14 PM Andy LoPresto  > wrote:
> Dan,
> 
> Does the proposal I submitted meet your requirements?
> 
> Andy LoPresto
> alopre...@apache.org 
> alopresto.apa...@gmail.com 
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> 
>> On Sep 5, 2018, at 4:09 PM, dan young > > wrote:
>> 
>> We're using it as well, in the same/similar fashion as being discussed in 
>> the thread...
>> 
>> Dano
>> 
>> On Wed, Sep 5, 2018, 10:07 AM Brandon DeVries > > wrote:
>> Andy,
>> 
>> We use it pretty much how Joe is... to create a unique composite key.  It 
>> seems as though that shouldn't be a difficult functionality to add.  
>> Possibly, you could flip your current dynamic key/value properties.  Make 
>> the key the name of the attribute you want to create, and the value is the 
>> attribute / attributes (newline delimited) that you want to include in the 
>> hash.  This does mean you can't use "${algorithm.name 
>> }" in the name of the created hash attribute, but I 
>> don't know if you'd consider that a big loss.  In any case, I'm sure there 
>> are other solutions, this is just a thought.
>> 
>> Brandon
>> 
>> On Wed, Sep 5, 2018 at 10:27 AM Joe Percivall > > wrote:
>> Hey Andy,
>> 
>> We're currently using the HashAttribute processor. The use-case is that we 
>> have various events that come in but sometimes those events are just updates 
>> of previous ones. We store everything in ElasticSearch. So for certain 
>> events, we'll calculate a hash based on a couple of attributes in order to 
>> have a composite unique key to upsert as the ES _id. This allows us to 
>> easily just insert/update events that are the same (as determined by the 
>> hashed composite key).
>> 
>> As for the configuration of the processors, we're essentially just 
>> specifying exact attributes as dynamic properties of HashAttribute. Then 
>> passing that FF to PutElasticSearchHttp with the resulting attribute from 
>> HashAttribute as the "Identifier Attribute".
>> 
>> Joe
>> 
>> On Mon, Sep 3, 2018 at 9:52 PM Andy LoPresto > > wrote:
>> I opened PRs for 2980 [1] and 2983 [2] which add more performant, 
>> consistent, and full-featured processors to calculate cryptographic hashes 
>> of flowfile content and flowfile attributes. I would like to deprecate and 
>> drop support for HashAttribute, as it performs a convoluted calculation that 
>> was probably useful in an old scenario, but doesn’t “hash attributes” like 
>> the name implies. As it blocks the new implementation from using that name 
>> and following our naming convention, I am hoping to find anyone still using 
>> the old implementation and understand their use case. Thanks for your help.
>> 
>> [1] https://github.com/apache/nifi/pull/2980 
>> 
>> [2] https://github.com/apache/nifi/pull/2983 
>> 
>> 
>> 
>> 
>> Andy LoPresto
>> alopre...@apache.org 
>> alopresto.apa...@gmail.com 
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>> 
>> 
>> 
>> --
>> Joe Percivall
>> linkedin.com/in/Percivall 
>> e: jperciv...@apache.com 



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Anyone using HashAttribute?

[dan young]

Heya Andy,

yes, that seems legit...we'll make it work on our side...

Keep up the awesome work on NiFi, powers all of our ETL here now :)

Dano

On Wed, Sep 5, 2018 at 5:14 PM Andy LoPresto  wrote:

> Dan,
>
> Does the proposal I submitted meet your requirements?
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Sep 5, 2018, at 4:09 PM, dan young  wrote:
>
> We're using it as well, in the same/similar fashion as being discussed in
> the thread...
>
> Dano
>
> On Wed, Sep 5, 2018, 10:07 AM Brandon DeVries  wrote:
>
>> Andy,
>>
>> We use it pretty much how Joe is... to create a unique composite key.  It
>> seems as though that shouldn't be a difficult functionality to add.
>> Possibly, you could flip your current dynamic key/value properties.  Make
>> the key the name of the attribute you want to create, and the value is the
>> attribute / attributes (newline delimited) that you want to include in the
>> hash.  This does mean you can't use "${algorithm.name}" in the name of
>> the created hash attribute, but I don't know if you'd consider that a big
>> loss.  In any case, I'm sure there are other solutions, this is just a
>> thought.
>>
>> Brandon
>>
>> On Wed, Sep 5, 2018 at 10:27 AM Joe Percivall 
>> wrote:
>>
>>> Hey Andy,
>>>
>>> We're currently using the HashAttribute processor. The use-case is that
>>> we have various events that come in but sometimes those events are just
>>> updates of previous ones. We store everything in ElasticSearch. So for
>>> certain events, we'll calculate a hash based on a couple of attributes in
>>> order to have a composite unique key to upsert as the ES _id. This allows
>>> us to easily just insert/update events that are the same (as determined by
>>> the hashed composite key).
>>>
>>> As for the configuration of the processors, we're essentially just
>>> specifying exact attributes as dynamic properties of HashAttribute. Then
>>> passing that FF to PutElasticSearchHttp with the resulting attribute from
>>> HashAttribute as the "Identifier Attribute".
>>>
>>> Joe
>>>
>>> On Mon, Sep 3, 2018 at 9:52 PM Andy LoPresto 
>>> wrote:
>>>
 I opened PRs for 2980 [1] and 2983 [2] which add more performant,
 consistent, and full-featured processors to calculate cryptographic hashes
 of flowfile content and flowfile attributes. I would like to deprecate and
 drop support for HashAttribute, as it performs a convoluted calculation
 that was probably useful in an old scenario, but doesn’t “hash attributes”
 like the name implies. As it blocks the new implementation from using that
 name and following our naming convention, I am hoping to find anyone still
 using the old implementation and understand their use case. Thanks for your
 help.

 [1] https://github.com/apache/nifi/pull/2980
 [2] https://github.com/apache/nifi/pull/2983



 Andy LoPresto
 alopre...@apache.org
 *alopresto.apa...@gmail.com *
 PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69


>>>
>>> --
>>> *Joe Percivall*
>>> linkedin.com/in/Percivall
>>> e: jperciv...@apache.com
>>>
>>
>

Re: Anyone using HashAttribute?

[Andy LoPresto]

Dan,

Does the proposal I submitted meet your requirements?

Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Sep 5, 2018, at 4:09 PM, dan young  wrote:
> 
> We're using it as well, in the same/similar fashion as being discussed in the 
> thread...
> 
> Dano
> 
> On Wed, Sep 5, 2018, 10:07 AM Brandon DeVries  > wrote:
> Andy,
> 
> We use it pretty much how Joe is... to create a unique composite key.  It 
> seems as though that shouldn't be a difficult functionality to add.  
> Possibly, you could flip your current dynamic key/value properties.  Make the 
> key the name of the attribute you want to create, and the value is the 
> attribute / attributes (newline delimited) that you want to include in the 
> hash.  This does mean you can't use "${algorithm.name 
> }" in the name of the created hash attribute, but I 
> don't know if you'd consider that a big loss.  In any case, I'm sure there 
> are other solutions, this is just a thought.
> 
> Brandon
> 
> On Wed, Sep 5, 2018 at 10:27 AM Joe Percivall  > wrote:
> Hey Andy,
> 
> We're currently using the HashAttribute processor. The use-case is that we 
> have various events that come in but sometimes those events are just updates 
> of previous ones. We store everything in ElasticSearch. So for certain 
> events, we'll calculate a hash based on a couple of attributes in order to 
> have a composite unique key to upsert as the ES _id. This allows us to easily 
> just insert/update events that are the same (as determined by the hashed 
> composite key).
> 
> As for the configuration of the processors, we're essentially just specifying 
> exact attributes as dynamic properties of HashAttribute. Then passing that FF 
> to PutElasticSearchHttp with the resulting attribute from HashAttribute as 
> the "Identifier Attribute".
> 
> Joe
> 
> On Mon, Sep 3, 2018 at 9:52 PM Andy LoPresto  > wrote:
> I opened PRs for 2980 [1] and 2983 [2] which add more performant, consistent, 
> and full-featured processors to calculate cryptographic hashes of flowfile 
> content and flowfile attributes. I would like to deprecate and drop support 
> for HashAttribute, as it performs a convoluted calculation that was probably 
> useful in an old scenario, but doesn’t “hash attributes” like the name 
> implies. As it blocks the new implementation from using that name and 
> following our naming convention, I am hoping to find anyone still using the 
> old implementation and understand their use case. Thanks for your help.
> 
> [1] https://github.com/apache/nifi/pull/2980 
> 
> [2] https://github.com/apache/nifi/pull/2983 
> 
> 
> 
> 
> Andy LoPresto
> alopre...@apache.org 
> alopresto.apa...@gmail.com 
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> 
> 
> 
> --
> Joe Percivall
> linkedin.com/in/Percivall 
> e: jperciv...@apache.com 


signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Anyone using HashAttribute?

[dan young]

We're using it as well, in the same/similar fashion as being discussed in
the thread...

Dano

On Wed, Sep 5, 2018, 10:07 AM Brandon DeVries  wrote:

> Andy,
>
> We use it pretty much how Joe is... to create a unique composite key.  It
> seems as though that shouldn't be a difficult functionality to add.
> Possibly, you could flip your current dynamic key/value properties.  Make
> the key the name of the attribute you want to create, and the value is the
> attribute / attributes (newline delimited) that you want to include in the
> hash.  This does mean you can't use "${algorithm.name}" in the name of
> the created hash attribute, but I don't know if you'd consider that a big
> loss.  In any case, I'm sure there are other solutions, this is just a
> thought.
>
> Brandon
>
> On Wed, Sep 5, 2018 at 10:27 AM Joe Percivall 
> wrote:
>
>> Hey Andy,
>>
>> We're currently using the HashAttribute processor. The use-case is that
>> we have various events that come in but sometimes those events are just
>> updates of previous ones. We store everything in ElasticSearch. So for
>> certain events, we'll calculate a hash based on a couple of attributes in
>> order to have a composite unique key to upsert as the ES _id. This allows
>> us to easily just insert/update events that are the same (as determined by
>> the hashed composite key).
>>
>> As for the configuration of the processors, we're essentially just
>> specifying exact attributes as dynamic properties of HashAttribute. Then
>> passing that FF to PutElasticSearchHttp with the resulting attribute from
>> HashAttribute as the "Identifier Attribute".
>>
>> Joe
>>
>> On Mon, Sep 3, 2018 at 9:52 PM Andy LoPresto 
>> wrote:
>>
>>> I opened PRs for 2980 [1] and 2983 [2] which add more performant,
>>> consistent, and full-featured processors to calculate cryptographic hashes
>>> of flowfile content and flowfile attributes. I would like to deprecate and
>>> drop support for HashAttribute, as it performs a convoluted calculation
>>> that was probably useful in an old scenario, but doesn’t “hash attributes”
>>> like the name implies. As it blocks the new implementation from using that
>>> name and following our naming convention, I am hoping to find anyone still
>>> using the old implementation and understand their use case. Thanks for your
>>> help.
>>>
>>> [1] https://github.com/apache/nifi/pull/2980
>>> [2] https://github.com/apache/nifi/pull/2983
>>>
>>>
>>>
>>> Andy LoPresto
>>> alopre...@apache.org
>>> *alopresto.apa...@gmail.com *
>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>>
>>>
>>
>> --
>> *Joe Percivall*
>> linkedin.com/in/Percivall
>> e: jperciv...@apache.com
>>
>

Re: Nifi Publish/Consumer Kafka and Azure Event Hub

[James Srinivasan]

I've not tried this myself, but once you have a working JAAS config
(from 
https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-quickstart-kafka-enabled-event-hubs#send-and-receive-messages-with-kafka-in-event-hubs),
set the corresponding protocol and mechanism properties in the NiFi
processor, and put the content of sasl.jaas.config in a file and
reference it from NiFi's bootstrap.conf as indicated by the second
link you found.

Good luck!
On Wed, 5 Sep 2018 at 15:54, João Henrique Freitas  wrote:
>
>
> Hello!
>
> I'm exploring Azure Event Hub with Kafka support. I know that's in preview.
>
> But I would like to know how to use PublishKafka with this configuration:
>
> https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-quickstart-kafka-enabled-event-hubs
>
> I don't know how to configure kafka processor with authentication parameters 
> like:
>
> security.protocol=SASL_SSL
> sasl.mechanism=PLAIN
> sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule 
> required username="$ConnectionString" 
> password="{YOUR.EVENTHUBS.CONNECTION.STRING}";
>
>
> Should I follow this?
>
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-kafka-1-0-nar/1.7.1/org.apache.nifi.processors.kafka.pubsub.PublishKafka_1_0/additionalDetails.html
>
> Or maybe its necessary do a patch in Publish/Consumer KafkaProcessor  ?
>
> Best regards,
>
> --
> João Henrique Ferreira de Freitas - joaohf_at_gmail.com
> Campinas-SP-Brasil

Re: parsing json array in ExecuteScript?

[Ed B]

I would suggest to use more convenient ways to achieve the same:

Split array into separate flow files, then use "EvaluateJsonPath" to
extract all the values into attributes (in case you know keys in advance).
If you don't know, when you can use first JoltTransformJSON, to generate
[{"name":"id","value":"10"},{"name":"info","value":"2am-3am"}.]
then you can split such arrays and evaluateJsonPath from "name" and "value"
into ff attributes.

On Wed, Sep 5, 2018 at 9:27 AM l vic  wrote:

> I have json array attribute:
> mylist
> :[{"id":10,"info":"2am-3am"},{"id":11,"info":"3AM-4AM"},{"id":12,"info":"4am-5am"}]
> How can I parse it to name/value pairs in groovy script?
> Thanks,
>

Re: PutSQL with multiple upserts?

[Ed B]

For PutSQL you have to generate a SQL statement.
For your previous question, I've suggested how to make that fork to
separate root attributes from array, and how to break array into separate
flow files (ff per element of array).
Now, having single elements, you can generate INSERT/UPDATE SQL statement
per each element and use PutSQL to execute it on your DB.
Alternatively, once you get separate flow files, you can use ConvertRecord
to convert your array elements into records, then use MergeRecords and then
use PutDatabaseRecord (that will improve performance because you can use
PutDatabaseRecord with multiple records in flow file).

On Wed, Sep 5, 2018 at 3:18 PM l vic  wrote:

> I need to save data with "root" object to be saved into "root" table and
> array "mylist" of contained objects to be saved into "mylist" table:
> {
> "id": 3,
> "name": "ROOT",
> "mylist": [{
> "id": 10,
> "info": "2am-3am"
> },
> {
> "id": 11,
> "info": "3AM-4AM"
> },
> {
> "id": 12,
> "info": "4am-5am"
> }]
>
> }
>
>  so I have to perform single "upsert" into "root" table and multiple
> "upsert"s into "mylist" table from one flow... Does PutSQL support this?
> Thank you
>
>

Re: Reporting connection timeout from InvokeHTTP Processor.

[Ed B]

Connection timeout is 504 response code.
InvokeHTTP will send FF to "Retry" relationship. You can use
RouteOnAttribute, check invokehttp.status.code attribute for having
appropriate code and then do your failure handling. You might also set "Always
Output Response" to "true".

On Wed, Sep 5, 2018 at 12:22 PM saloni udani 
wrote:

> Hi
>
> Is there a way to generate a failure flow file for connection issues like
> connection timeout for InvokeHTTP Processor?
>
>
>
>
>
> Thanks
> Saloni Udani
>

PutSQL with multiple upserts?

[l vic]

I need to save data with "root" object to be saved into "root" table and
array "mylist" of contained objects to be saved into "mylist" table:
{
"id": 3,
"name": "ROOT",
"mylist": [{
"id": 10,
"info": "2am-3am"
},
{
"id": 11,
"info": "3AM-4AM"
},
{
"id": 12,
"info": "4am-5am"
}]

}

 so I have to perform single "upsert" into "root" table and multiple
"upsert"s into "mylist" table from one flow... Does PutSQL support this?
Thank you

Re: Anyone using HashAttribute?

[Andy LoPresto]

Joe and Brandon,

Thanks for your input here. I agree that changing the behavior of an existing 
processor (that is used in people’s flows) is a breaking change and probably 
requires a major release, which is why I didn’t do that in the PRs. As written 
today, they are fully backward-compatible. My concern is that users have had 
issues because they attempt to deploy HashAttribute expecting it to perform 
hashing of individual attributes. The introduction of 
CryptographicHashAttribute provides this functionality but the 
discoverability/“do what the name says” issue feels to me like a new addition 
to the “death by 1000 paper cuts” list that any complex project like NiFi has 
to endure.

The CryptographicHashContent processor is fully backward-compatible on 
(expected) functionality, but because of the property descriptor naming, it’s 
not “in-place” replaceable. Instead, (legacy) HashContent is marked as 
deprecated, and moving forward, CHC should be used.

Given your well-described use cases for HA, I think I may be able to provide 
that in CHA as well. I would expect to add a dropdown PD for “attribute 
enumeration style” and offer “individual” (each hash is generated on a single 
attribute), “list” (each hash is generated over an ordered, delimited list of 
literal matches), and “regex” (each hash is generated over an ordered list of 
all attribute names matching the provided regex). Then the dynamic properties 
would describe the output, as happens in the existing PR. Maybe a custom 
delimiter property is needed too, but for now ‘’ could be used to join the 
values. I’ll write up a Jira for this, and hopefully you can both let me know 
if this meets your requirements.

Example:

*Incoming Flowfile*

attributes: [username: “alopresto”, role: “security”, email: 
“alopre...@apache.org ”, git_account: “alopresto”]

*CHA Properties (Individual)*

attribute_enumeration_style: “individual”
(dynamic) username_sha256: “username”
(dynamic) git_account_sha256: “git_account”

*Behavior (Individual)*

username_sha256 = git_account_sha256 = $(echo -n "alopresto" | shasum -a 256) = 
600973dc8f2b7bb2a20651ebefe4bf91c5295afef19f4d5b9994d581f5a68a23

*Resulting Flowfile (Individual)*

attributes: [username: “alopresto”, role: “security”, email: 
“alopre...@apache.org ”, git_account: “alopresto”, 
username_sha256: 
“600973dc8f2b7bb2a20651ebefe4bf91c5295afef19f4d5b9994d581f5a68a23”, 
git_account_sha256: 
“600973dc8f2b7bb2a20651ebefe4bf91c5295afef19f4d5b9994d581f5a68a23"]

*CHA Properties (List)*

attribute_enumeration_style: “list”
(dynamic) username_and_email_sha256: “username, email”
(dynamic) git_account_sha256: “git_account”

*Behavior (List)*

username_and_email_sha256 = $(echo -n "aloprestoalopre...@apache.org" | shasum 
-a 256) = 22a11b7b3173f95c23a1f434949ec2a2e66455b9cb26b7ebc90afca25d91333f
git_account_sha256 = $(echo -n "alopresto" | shasum -a 256) = 
600973dc8f2b7bb2a20651ebefe4bf91c5295afef19f4d5b9994d581f5a68a23

*Resulting Flowfile (List)*

attributes: [username: “alopresto”, role: “security”, email: 
“alopre...@apache.org ”, git_account: “alopresto”, 
username_email_sha256: “ 
22a11b7b3173f95c23a1f434949ec2a2e66455b9cb26b7ebc90afca25d91333f”, 
git_account_sha256: 
“600973dc8f2b7bb2a20651ebefe4bf91c5295afef19f4d5b9994d581f5a68a23”]

*CHA Properties (Regex)*

attribute_enumeration_style: “regex”
(dynamic) all_sha256: “.*”
(dynamic) git_account_sha256: “git_account”

*Behavior (Regex)*

all_sha256 = sort(attributes_that_match_regex) = [email, git_account, role, 
username] = $(echo -n "alopresto@apache.orgaloprestosecurityalopresto" | shasum 
-a 256) = b370fdf0132933cea76e3daa3d4a437bb8c571dd0cd0e79ee5d7759cf64efced
git_account_sha256 = $(echo -n "alopresto" | shasum -a 256) = 
600973dc8f2b7bb2a20651ebefe4bf91c5295afef19f4d5b9994d581f5a68a23

*Resulting Flowfile (Regex)*

attributes: [username: “alopresto”, role: “security”, email: 
“alopre...@apache.org ”, git_account: “alopresto”, 
all_sha256: “ 
b370fdf0132933cea76e3daa3d4a437bb8c571dd0cd0e79ee5d7759cf64efced”, 
git_account_sha256: 
“600973dc8f2b7bb2a20651ebefe4bf91c5295afef19f4d5b9994d581f5a68a23”]

Mike,

I don’t think it makes sense to remove this functionality and relocate it to 
Elasticsearch. Adding a capacity to calculate a unique identifier over multiple 
inputs may be valuable for Elasticsearch specifically, but the functionality 
described here is independent from that use case, and as NiFi’s processor 
design philosophy is similar to *nix builtins (do one thing; do it well), it 
makes more sense to chain the individual necessary processors.


Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Sep 5, 2018, at 9:33 AM, Brandon DeVries  wrote:
> 
> Mike,
> 
> We don't use it with Elasticsearch.
> 
> Fundamentally, it feels like the problem is that 

Re: Anyone using HashAttribute?

[Juan Pablo Gardella]

I vote to keep for backward compatibility.

On Wed, 5 Sep 2018 at 13:33 Brandon DeVries  wrote:

> Mike,
>
> We don't use it with Elasticsearch.
>
> Fundamentally, it feels like the problem is that this change would break
> backwards compatibility, which would require a major version bump.  So, in
> lieu of that, the options are probably 1) use a different name or 2) put
> the new functionality in HashContent as something that can be toggled on,
> but leaving the current behavior as the default.
>
> Brandon
>
> On Wed, Sep 5, 2018 at 12:21 PM Mike Thomsen 
> wrote:
>
>> Brandon,
>>
>> What processor do you use it for in that capacity? If it's an
>> ElasticSearch one we can look into ways to bring this functionality into
>> that bundle so Andy can refactor.
>>
>> Thanks,
>>
>> Mike
>>
>> On Wed, Sep 5, 2018 at 12:07 PM Brandon DeVries  wrote:
>>
>>> Andy,
>>>
>>> We use it pretty much how Joe is... to create a unique composite key.
>>> It seems as though that shouldn't be a difficult functionality to add.
>>> Possibly, you could flip your current dynamic key/value properties.  Make
>>> the key the name of the attribute you want to create, and the value is the
>>> attribute / attributes (newline delimited) that you want to include in the
>>> hash.  This does mean you can't use "${algorithm.name}" in the name of
>>> the created hash attribute, but I don't know if you'd consider that a big
>>> loss.  In any case, I'm sure there are other solutions, this is just a
>>> thought.
>>>
>>> Brandon
>>>
>>> On Wed, Sep 5, 2018 at 10:27 AM Joe Percivall 
>>> wrote:
>>>
 Hey Andy,

 We're currently using the HashAttribute processor. The use-case is that
 we have various events that come in but sometimes those events are just
 updates of previous ones. We store everything in ElasticSearch. So for
 certain events, we'll calculate a hash based on a couple of attributes in
 order to have a composite unique key to upsert as the ES _id. This allows
 us to easily just insert/update events that are the same (as determined by
 the hashed composite key).

 As for the configuration of the processors, we're essentially just
 specifying exact attributes as dynamic properties of HashAttribute. Then
 passing that FF to PutElasticSearchHttp with the resulting attribute from
 HashAttribute as the "Identifier Attribute".

 Joe

 On Mon, Sep 3, 2018 at 9:52 PM Andy LoPresto 
 wrote:

> I opened PRs for 2980 [1] and 2983 [2] which add more performant,
> consistent, and full-featured processors to calculate cryptographic hashes
> of flowfile content and flowfile attributes. I would like to deprecate and
> drop support for HashAttribute, as it performs a convoluted calculation
> that was probably useful in an old scenario, but doesn’t “hash attributes”
> like the name implies. As it blocks the new implementation from using that
> name and following our naming convention, I am hoping to find anyone still
> using the old implementation and understand their use case. Thanks for 
> your
> help.
>
> [1] https://github.com/apache/nifi/pull/2980
> [2] https://github.com/apache/nifi/pull/2983
>
>
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
>

 --
 *Joe Percivall*
 linkedin.com/in/Percivall
 e: jperciv...@apache.com

>>>

Re: Anyone using HashAttribute?

[Brandon DeVries]

Mike,

We don't use it with Elasticsearch.

Fundamentally, it feels like the problem is that this change would break
backwards compatibility, which would require a major version bump.  So, in
lieu of that, the options are probably 1) use a different name or 2) put
the new functionality in HashContent as something that can be toggled on,
but leaving the current behavior as the default.

Brandon

On Wed, Sep 5, 2018 at 12:21 PM Mike Thomsen  wrote:

> Brandon,
>
> What processor do you use it for in that capacity? If it's an
> ElasticSearch one we can look into ways to bring this functionality into
> that bundle so Andy can refactor.
>
> Thanks,
>
> Mike
>
> On Wed, Sep 5, 2018 at 12:07 PM Brandon DeVries  wrote:
>
>> Andy,
>>
>> We use it pretty much how Joe is... to create a unique composite key.  It
>> seems as though that shouldn't be a difficult functionality to add.
>> Possibly, you could flip your current dynamic key/value properties.  Make
>> the key the name of the attribute you want to create, and the value is the
>> attribute / attributes (newline delimited) that you want to include in the
>> hash.  This does mean you can't use "${algorithm.name}" in the name of
>> the created hash attribute, but I don't know if you'd consider that a big
>> loss.  In any case, I'm sure there are other solutions, this is just a
>> thought.
>>
>> Brandon
>>
>> On Wed, Sep 5, 2018 at 10:27 AM Joe Percivall 
>> wrote:
>>
>>> Hey Andy,
>>>
>>> We're currently using the HashAttribute processor. The use-case is that
>>> we have various events that come in but sometimes those events are just
>>> updates of previous ones. We store everything in ElasticSearch. So for
>>> certain events, we'll calculate a hash based on a couple of attributes in
>>> order to have a composite unique key to upsert as the ES _id. This allows
>>> us to easily just insert/update events that are the same (as determined by
>>> the hashed composite key).
>>>
>>> As for the configuration of the processors, we're essentially just
>>> specifying exact attributes as dynamic properties of HashAttribute. Then
>>> passing that FF to PutElasticSearchHttp with the resulting attribute from
>>> HashAttribute as the "Identifier Attribute".
>>>
>>> Joe
>>>
>>> On Mon, Sep 3, 2018 at 9:52 PM Andy LoPresto 
>>> wrote:
>>>
 I opened PRs for 2980 [1] and 2983 [2] which add more performant,
 consistent, and full-featured processors to calculate cryptographic hashes
 of flowfile content and flowfile attributes. I would like to deprecate and
 drop support for HashAttribute, as it performs a convoluted calculation
 that was probably useful in an old scenario, but doesn’t “hash attributes”
 like the name implies. As it blocks the new implementation from using that
 name and following our naming convention, I am hoping to find anyone still
 using the old implementation and understand their use case. Thanks for your
 help.

 [1] https://github.com/apache/nifi/pull/2980
 [2] https://github.com/apache/nifi/pull/2983



 Andy LoPresto
 alopre...@apache.org
 *alopresto.apa...@gmail.com *
 PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69


>>>
>>> --
>>> *Joe Percivall*
>>> linkedin.com/in/Percivall
>>> e: jperciv...@apache.com
>>>
>>

Reporting connection timeout from InvokeHTTP Processor.

[saloni udani]

Hi

Is there a way to generate a failure flow file for connection issues like
connection timeout for InvokeHTTP Processor?





Thanks
Saloni Udani

Re: Anyone using HashAttribute?

[Mike Thomsen]

Brandon,

What processor do you use it for in that capacity? If it's an ElasticSearch
one we can look into ways to bring this functionality into that bundle so
Andy can refactor.

Thanks,

Mike

On Wed, Sep 5, 2018 at 12:07 PM Brandon DeVries  wrote:

> Andy,
>
> We use it pretty much how Joe is... to create a unique composite key.  It
> seems as though that shouldn't be a difficult functionality to add.
> Possibly, you could flip your current dynamic key/value properties.  Make
> the key the name of the attribute you want to create, and the value is the
> attribute / attributes (newline delimited) that you want to include in the
> hash.  This does mean you can't use "${algorithm.name}" in the name of
> the created hash attribute, but I don't know if you'd consider that a big
> loss.  In any case, I'm sure there are other solutions, this is just a
> thought.
>
> Brandon
>
> On Wed, Sep 5, 2018 at 10:27 AM Joe Percivall 
> wrote:
>
>> Hey Andy,
>>
>> We're currently using the HashAttribute processor. The use-case is that
>> we have various events that come in but sometimes those events are just
>> updates of previous ones. We store everything in ElasticSearch. So for
>> certain events, we'll calculate a hash based on a couple of attributes in
>> order to have a composite unique key to upsert as the ES _id. This allows
>> us to easily just insert/update events that are the same (as determined by
>> the hashed composite key).
>>
>> As for the configuration of the processors, we're essentially just
>> specifying exact attributes as dynamic properties of HashAttribute. Then
>> passing that FF to PutElasticSearchHttp with the resulting attribute from
>> HashAttribute as the "Identifier Attribute".
>>
>> Joe
>>
>> On Mon, Sep 3, 2018 at 9:52 PM Andy LoPresto 
>> wrote:
>>
>>> I opened PRs for 2980 [1] and 2983 [2] which add more performant,
>>> consistent, and full-featured processors to calculate cryptographic hashes
>>> of flowfile content and flowfile attributes. I would like to deprecate and
>>> drop support for HashAttribute, as it performs a convoluted calculation
>>> that was probably useful in an old scenario, but doesn’t “hash attributes”
>>> like the name implies. As it blocks the new implementation from using that
>>> name and following our naming convention, I am hoping to find anyone still
>>> using the old implementation and understand their use case. Thanks for your
>>> help.
>>>
>>> [1] https://github.com/apache/nifi/pull/2980
>>> [2] https://github.com/apache/nifi/pull/2983
>>>
>>>
>>>
>>> Andy LoPresto
>>> alopre...@apache.org
>>> *alopresto.apa...@gmail.com *
>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>>
>>>
>>
>> --
>> *Joe Percivall*
>> linkedin.com/in/Percivall
>> e: jperciv...@apache.com
>>
>

Re: Anyone using HashAttribute?

[Brandon DeVries]

Andy,

We use it pretty much how Joe is... to create a unique composite key.  It
seems as though that shouldn't be a difficult functionality to add.
Possibly, you could flip your current dynamic key/value properties.  Make
the key the name of the attribute you want to create, and the value is the
attribute / attributes (newline delimited) that you want to include in the
hash.  This does mean you can't use "${algorithm.name}" in the name of the
created hash attribute, but I don't know if you'd consider that a big
loss.  In any case, I'm sure there are other solutions, this is just a
thought.

Brandon

On Wed, Sep 5, 2018 at 10:27 AM Joe Percivall  wrote:

> Hey Andy,
>
> We're currently using the HashAttribute processor. The use-case is that we
> have various events that come in but sometimes those events are just
> updates of previous ones. We store everything in ElasticSearch. So for
> certain events, we'll calculate a hash based on a couple of attributes in
> order to have a composite unique key to upsert as the ES _id. This allows
> us to easily just insert/update events that are the same (as determined by
> the hashed composite key).
>
> As for the configuration of the processors, we're essentially just
> specifying exact attributes as dynamic properties of HashAttribute. Then
> passing that FF to PutElasticSearchHttp with the resulting attribute from
> HashAttribute as the "Identifier Attribute".
>
> Joe
>
> On Mon, Sep 3, 2018 at 9:52 PM Andy LoPresto  wrote:
>
>> I opened PRs for 2980 [1] and 2983 [2] which add more performant,
>> consistent, and full-featured processors to calculate cryptographic hashes
>> of flowfile content and flowfile attributes. I would like to deprecate and
>> drop support for HashAttribute, as it performs a convoluted calculation
>> that was probably useful in an old scenario, but doesn’t “hash attributes”
>> like the name implies. As it blocks the new implementation from using that
>> name and following our naming convention, I am hoping to find anyone still
>> using the old implementation and understand their use case. Thanks for your
>> help.
>>
>> [1] https://github.com/apache/nifi/pull/2980
>> [2] https://github.com/apache/nifi/pull/2983
>>
>>
>>
>> Andy LoPresto
>> alopre...@apache.org
>> *alopresto.apa...@gmail.com *
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>
>>
>
> --
> *Joe Percivall*
> linkedin.com/in/Percivall
> e: jperciv...@apache.com
>

Nifi Publish/Consumer Kafka and Azure Event Hub

[João Henrique Freitas]

Hello!

I'm exploring Azure Event Hub with Kafka support. I know that's in preview.

But I would like to know how to use PublishKafka with this configuration:

https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-quickstart-kafka-enabled-event-hubs

I don't know how to configure kafka processor with authentication
parameters like:

security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule
required username="$ConnectionString"
password="{YOUR.EVENTHUBS.CONNECTION.STRING}";


Should I follow this?

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-kafka-1-0-nar/1.7.1/org.apache.nifi.processors.kafka.pubsub.PublishKafka_1_0/additionalDetails.html

Or maybe its necessary do a patch in Publish/Consumer KafkaProcessor  ?

Best regards,

-- 
João Henrique Ferreira de Freitas - joaohf_at_gmail.com
Campinas-SP-Brasil

Re: Anyone using HashAttribute?

[Joe Percivall]

Hey Andy,

We're currently using the HashAttribute processor. The use-case is that we
have various events that come in but sometimes those events are just
updates of previous ones. We store everything in ElasticSearch. So for
certain events, we'll calculate a hash based on a couple of attributes in
order to have a composite unique key to upsert as the ES _id. This allows
us to easily just insert/update events that are the same (as determined by
the hashed composite key).

As for the configuration of the processors, we're essentially just
specifying exact attributes as dynamic properties of HashAttribute. Then
passing that FF to PutElasticSearchHttp with the resulting attribute from
HashAttribute as the "Identifier Attribute".

Joe

On Mon, Sep 3, 2018 at 9:52 PM Andy LoPresto  wrote:

> I opened PRs for 2980 [1] and 2983 [2] which add more performant,
> consistent, and full-featured processors to calculate cryptographic hashes
> of flowfile content and flowfile attributes. I would like to deprecate and
> drop support for HashAttribute, as it performs a convoluted calculation
> that was probably useful in an old scenario, but doesn’t “hash attributes”
> like the name implies. As it blocks the new implementation from using that
> name and following our naming convention, I am hoping to find anyone still
> using the old implementation and understand their use case. Thanks for your
> help.
>
> [1] https://github.com/apache/nifi/pull/2980
> [2] https://github.com/apache/nifi/pull/2983
>
>
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
>

-- 
*Joe Percivall*
linkedin.com/in/Percivall
e: jperciv...@apache.com

Re: parsing json array in ExecuteScript?

[Mike Thomsen]

> How can I parse it to name/value pairs in groovy script?

I would recommend getting the Groovy binary distribution (we use 2.4.X) and
experimenting with that. Aside from us throwing in a few of the NiFi APIs,
it's a standard Groovy environment. You'll flatten the learning curve on
writing these scripts if you approach it that way.

Thanks,

Mike

On Wed, Sep 5, 2018 at 9:34 AM Matt Burgess  wrote:

> Are you trying to collect them into a single data structure, or just
> iterate over the key/value pairs? For the former, you'd need to change
> the key names somehow, since each object in the array has the same
> keys. For the latter, you can use each() to iterate over any
> collection:
>
> mylist.each {arrayEntry -> arrayEntry.each {k,v -> println "$k = $v"}}
>
> Note that the each() for a Map allows you to get the key and value
> separately in the closure, rather than iterating over Map.Entry
> objects and having to call it.key and it.value.
>
> Regards,
> Matt
> On Wed, Sep 5, 2018 at 9:27 AM l vic  wrote:
> >
> > I have json array attribute:
> > mylist
> :[{"id":10,"info":"2am-3am"},{"id":11,"info":"3AM-4AM"},{"id":12,"info":"4am-5am"}]
> > How can I parse it to name/value pairs in groovy script?
> > Thanks,
>

unsubscribe

[Rick]

unsubscribe

parsing json array in ExecuteScript?

[l vic]

I have json array attribute:
mylist
:[{"id":10,"info":"2am-3am"},{"id":11,"info":"3AM-4AM"},{"id":12,"info":"4am-5am"}]
How can I parse it to name/value pairs in groovy script?
Thanks,

A sensible approach to scheduling via the API?

[Vos, Walter]

Hi,

In our big data environment one of the architectural principles is to schedule 
jobs with Azure Automation (runbooks). A scheduling database is used to decide 
when to start which jobs. NiFi flows however are currently being scheduled in 
NiFi itself. We're looking for a good approach to move this over to runbooks. I 
see a couple of options:

* Have each flow start with a timer driven processor, where the run schedule is 
an hour or so. This processor will be stopped by default, and can be turned on 
via the API. It is then stopped at some point before the run schedule ends, 
preventing the processor from running twice.
* Use a ListenHTTP processor that we can POST a message to that specifies which 
flow to start. Do something like RouteOnAttribute to choose the right flow. I 
imagine this as being one ListenHTTP processor that is connected to all flows.
* Translate the schedule from the scheduling database to a ChronTrigger 
expression. Check if the CRON schedule on the processor is indeed set to that 
schedule. If not, stop the processor, change the schedule and start it again. 
If it is, do nothing and assume it'll run. This one seems convoluted on the one 
hand, but requires the least architecture within NiFi itself I imagine.

What do you think? Has anyone had to deal with something like this? How did you 
solve it? I can't find much information about this on the web, although I could 
be using the wrong terms.

Kind regards,

Walter Vos




Deze e-mail, inclusief eventuele bijlagen, is uitsluitend bestemd voor (gebruik 
door) de geadresseerde. De e-mail kan persoonlijke of vertrouwelijke informatie 
bevatten. Openbaarmaking, vermenigvuldiging, verspreiding en/of verstrekking 
van (de inhoud van) deze e-mail (en eventuele bijlagen) aan derden is 
uitdrukkelijk niet toegestaan. Indien u niet de bedoelde geadresseerde bent, 
wordt u vriendelijk verzocht degene die de e-mail verzond hiervan direct op de 
hoogte te brengen en de e-mail (en eventuele bijlagen) te vernietigen.

Informatie vennootschap