Re: How to find average of two lines in NiFi?

[Andy LoPresto]

Prabhu,

I don’t think NiFi is the right tool to do the complete task (fine for CSV 
parsing & splitting), but you should look at writing a script or using Spark, 
etc. to do the data operations. Any NiFi-native solution will be brittle and 
won’t scale well.

That said, I did answer how you could do this in NiFi only on your 
StackOverflow question [1].

However, you could certainly use a combination of SplitText processors to get 
the proper data into individual flowfiles (i.e. all Mahi rows in one, all Andy 
rows in another). Once you have a record that looks like:

Andy,1,85
Andy,1,95
you can use ExtractText with regular expressions to get 85 and 95 into 
attributes marks.1 and marks.2 (a good example of where scaling will break down 
-- doing this with 2 rows is easy; doing this with 100k is ridiculous). You can 
then use UpdateAttribute with the Expression Language to calculate the average 
of those two attributes (convert toNumber() first) and populate a third 
attribute marks.average (either through chaining plus() and divide() functions 
or with the math advanced operation 

 (uses Java Reflection)). Once you have the desired result in an attribute, use 
ReplaceText to update the flowfile content, and MergeContent to merge the 
individual flowfiles back into a single instance.




[1] http://stackoverflow.com/a/42381999/70465


Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Feb 21, 2017, at 5:20 AM, prabhu Mahendran  wrote:
> 
> i need to find average of two values in seperate lines.
> 
> My Csv file loo like this.,
> 
> Name,ID,Marks
> Mahi,1,90
> Mahi,1,90
> 
> 
> Andy,2,100
> Andy,2,100
> Now i need to store that average of 2 marks in database. "Average" column 
> should add two marks and divide with 2 and store that result in SQL query
> 
> Table:
> 
> Name,ID,Average
> Mahi,2,90
> Andy,2,100
> Is this possible to find average of two value's in seperate rows using NiFi?
> 



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: ConsumeKafka processor erroring when held up by full queue

[Koji Kawamura]

Hi Nick,

I understand that. I will continue adding more code to iron out implementation.
Please let me know if you find anything by looking at the code. I'd
like you to review the branch in detail once the PR is ready.

Thanks!
Koji

On Tue, Feb 21, 2017 at 9:06 AM, Nick Carenza
 wrote:
> Hey Koji, thanks for putting in the time.
>
> I have not had a chance to start working on this myself and I certainly
> support any effort to resolve it. I'll take a look at your branch and play
> around with it.
>
> Thanks!,
> Nick
>
> On Mon, Feb 20, 2017 at 2:30 AM, Koji Kawamura 
> wrote:
>>
>> Hi Nick, Joe, Bryan,
>>
>> I confirmed that this is easily reproducible and I got exactly the
>> same stacktrace.
>>
>> Also I was curious about how Kafka consumer's pause/resume API works,
>> so I've gone further and done a simple experiment to see if it helps
>> in this situation to retain consumer connection.
>>
>> The experimentation code in my remote branch here.
>>
>> https://github.com/ijokarumawak/nifi/commit/28ba134771ec7a7e810924f655662a29662ba9bf
>>
>> It seems working as expected. After back-pressure is engaged,
>> ConsumeKafka is not triggered for a while, then when downstream
>> recovers from back-pressure, ConsumeKafka resumes consuming messages
>> without any loss or warning messages using the same consumer instance.
>>
>> Nick, have you already start working on fixing this?
>> If you haven't, and the approach using pause/resume sounds reasonable,
>> I am going to add more synchronization, start/stop processor handling,
>> and make it work more properly with onTrigger so that it only try to
>> retain when there hasn't been any polling activity for some period of
>> time.
>>
>> How do you think?
>> Any feedback would be appreciated. Thanks!
>>
>> Koji
>>
>> On Fri, Feb 10, 2017 at 7:22 AM, Nick Carenza
>>  wrote:
>> > Thank you guys, I will look to see what I can do to contribute.
>> >
>> > On Thu, Feb 9, 2017 at 1:19 PM, Joe Witt  wrote:
>> >>
>> >> That said I think we can improve our handling of the consumer (kafka
>> >> client) and session (nifi transactional logic) and solve the problem.
>> >> It is related to our backpressure/consumer handling so we can fix
>> >> that.
>> >>
>> >> Thanks
>> >> Joe
>> >>
>> >> On Thu, Feb 9, 2017 at 1:38 PM, Bryan Bende  wrote:
>> >> > No data loss, but you may process the same message twice in NiFi.
>> >> >
>> >> > The ordering of operations is:
>> >> >
>> >> > 1) poll Kafka
>> >> > 2) write received data to flow file
>> >> > 3) commit NiFi session so data in flow file cannot be lost
>> >> > 4) commit offsets to Kafka
>> >> >
>> >> > Doing it this way achieves at-least once processing which means you
>> >> > can't ever lose data, but you can process data twice.
>> >> >
>> >> > If we committed the offsets before committing the flow file you would
>> >> > never get duplicates, but you could lose a message if a crash
>> >> > happened
>> >> > between commit the offset and committing the NiFi session (at-most
>> >> > once processing).
>> >> >
>> >> > So the error is happening on #4 and NiFi has already produced a flow
>> >> > file with the message, but then Kafka says it can't update the
>> >> > offset,
>> >> > and then another consumer will likely pull that same message again
>> >> > and
>> >> > produce another flow file with the same message.
>> >> >
>> >> >
>> >> > On Thu, Feb 9, 2017 at 1:19 PM, Nick Carenza
>> >> >  wrote:
>> >> >> That makes perfect sense. To be clear, is there any potential to
>> >> >> lose
>> >> >> messages in this scenario?
>> >> >>
>> >> >> On Thu, Feb 9, 2017 at 7:16 AM, Joe Witt  wrote:
>> >> >>>
>> >> >>> yeah this is probably a good case/cause for use of the pause
>> >> >>> concept
>> >> >>> in kafka consumers.
>> >> >>>
>> >> >>> On Thu, Feb 9, 2017 at 9:49 AM, Bryan Bende 
>> >> >>> wrote:
>> >> >>> > I believe you are running into this issue:
>> >> >>> >
>> >> >>> > https://issues.apache.org/jira/browse/NIFI-3189
>> >> >>> >
>> >> >>> > When back-pressure happens on the queue coming out of
>> >> >>> > ConsumeKafka,
>> >> >>> > this can last for longer than session.timeout.ms, and when the
>> >> >>> > processors resumes executing it receives this error on the first
>> >> >>> > execution. We should be able to implement some type of keep-alive
>> >> >>> > so
>> >> >>> > that even when the processor is not executing, there is a
>> >> >>> > background
>> >> >>> > thread, or some way of keeping the connections alive.
>> >> >>> >
>> >> >>> > I believe any user-defined properties in the processor get passed
>> >> >>> > to
>> >> >>> > the Kafka consumer, so I believe you could add
>> >> >>> > "session.timeout.ms"
>> >> >>> > and set a much higher value as a possible work around.
>> >> >>> >
>> >> >>> > Thanks,
>> >> >>> >
>> >> >>> > Bryan
>> >> >>> >
>> >> >>> > On Thu, Feb 9, 2017 at 8:42 AM, Koji Kawamura
>> >> >>> > 
>> >> >>> > wrote:
>> >> >>> >> Hello Nick,
>> >> >>> >>
>> >> >>> >> First, I assume "had a queue back up" means have a queue being
>

Re: NiFi Users: Powered by NiFi page

[Joe Witt]

nope you did just fine.  i'll try to merge it now

On Tue, Feb 21, 2017 at 4:01 PM, Jagrut Sharma  wrote:
> Hi - I've submitted a pull request to 'nifi-site' for requesting inclusion
> of Think Big, A Teradata Company in the powered-by NiFi page. Please let me
> know if I should provide them via an email to this list also.
>
> Thanks.
> --
> Jagrut
>
> On Tue, Feb 21, 2017 at 12:06 PM, Andy LoPresto 
> wrote:
>>
>> Adding Hashmap, Inc. from Chris Herrera:
>>
>> Begin forwarded message:
>>
>> From: Chris Herrera 
>> Subject: Re: Powered by NiFi
>> Date: February 21, 2017 at 12:00:15 PM PST
>> To: Andy LoPresto 
>>
>> No problem at all!
>>
>> Regards,
>> Chris
>>
>> On Feb 21, 2017, 1:59 PM -0600, Andy LoPresto ,
>> wrote:
>>
>> Thanks Chris. Mind if I forward this last message to the list in order to
>> have a record of your approval?
>>
>> Andy LoPresto
>> alopre...@apache.org
>> alopresto.apa...@gmail.com
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>
>> On Feb 21, 2017, at 11:52 AM, Chris Herrera 
>> wrote:
>>
>> Hey Andy,
>>
>> Sorry it took a bit to get back to you…we would love to be able to put
>> something on the powered by nifi page:
>>
>> Company: Hashmap, Inc. (www.hashmapinc.com)
>>
>> Industry: Big Data / IoT
>>
>> Summary:
>> Hashmap uses Apache NiFi to securely collect, transmit, and transform data
>> for ingest and delivery into our IoT / Time Series Accelerator platform,
>> allowing for outcome-based, real time analytics and visualization of oil &
>> gas, utilities, manufacturing, industrial, retail, pharma, and process
>> control data. Additionally, we are creating a catalog of open source,
>> ready-to-run, industry specific NiFi processors and controller services for
>> protocols like OPC-UA, ETP, WITSML, LAS, and many others.
>>
>> Regards,
>> Chris
>>
>>
>> Andy LoPresto
>> alopre...@apache.org
>> alopresto.apa...@gmail.com
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>
>> On Feb 16, 2017, at 2:39 PM, Andre  wrote:
>>
>> wow! This is looker of LookML fame. Nice one!
>>
>> I remember watching this video
>>
>> https://youtu.be/8LqnC9IjPYY
>>
>> about their platform a long time ago...
>>
>>
>>
>> On Fri, Feb 17, 2017 at 8:10 AM, Andy LoPresto 
>> wrote:
>>>
>>> Documentation that this was ok:
>>>
>>> Begin forwarded message:
>>>
>>> From: dan young 
>>> Subject: Re: Apache NiFi + Looker
>>> Date: February 16, 2017 at 11:59:16 AM PST
>>> To: Andy LoPresto 
>>>
>>> Sure, i think that's fine.
>>>
>>> On Thu, Feb 16, 2017 at 12:12 PM Andy LoPresto 
>>> wrote:

 Dan,

 Do I have your permission to forward the relevant part of this exchange
 to the users@ list in order to document that you provided this info? 
 Thanks.

 Andy LoPresto
 alopre...@apache.org
 alopresto.apa...@gmail.com
 PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

 On Feb 16, 2017, at 10:45 AM, Andy LoPresto 
 wrote:

 Got it. Thanks.

 Andy LoPresto
 alopre...@apache.org
 alopresto.apa...@gmail.com
 PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

 On Feb 16, 2017, at 10:34 AM, dan young  wrote:

 We're a SaaS, anaylitics software...

 On Thu, Feb 16, 2017, 11:34 AM Andy LoPresto 
 wrote:
>
> This is perfect, Dan. For “industry”, would you say “Data Consulting”
> or do you have a better term? I’ll add this to the site. Thanks.
>
> Andy LoPresto
> alopre...@apache.org
> alopresto.apa...@gmail.com
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Feb 16, 2017, at 10:31 AM, dan young  wrote:
>
> Hey Andy,
>
> Got the green light to add us to the Powered by NiFi.
>
>
> Our website:  https://looker.com/
>
> Blurb: We're building all new data ingestion pipelines using NiFi.
> Existing pipeline are being migrated to NiFi as well. We have deployed 
> NiFi
> clusters to ingest, transform and deliver data to various backends like
> Google Big Query, Amazon Redshift, and Amazon S3.
>
> LMK if you have any questions and/or need something different.
>
> Dano
>>>
>>>
>>> Andy LoPresto
>>> alopre...@apache.org
>>> alopresto.apa...@gmail.com
>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>>
>>> On Feb 16, 2017, at 11:05 AM, Andy LoPresto  wrote:
>>>
>>> Added Looker from Dan Young.
>>>
>>> Andy LoPresto
>>> alopre...@apache.org
>>> alopresto.apa...@gmail.com
>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>>
>>> On Feb 15, 2017, at 8:58 AM, Aldrin Piri  wrote:
>>>
>>> Hi Giovanni,
>>>
>>> GoDataDriven has been included.
>>>
>>> Thanks!
>>>
>>> On Wed, Feb 15, 2017 at 11:48 AM, Giovanni Lanzani
>>>  wrote:

 Hi Joe,

 You can put GoDataDriven (https://godatadriven.com)
 Summary: GoDataDriven, a Dutch service company in the data science an

Re: NiFi Users: Powered by NiFi page

[Jagrut Sharma]

Hi - I've submitted a pull request to 'nifi-site' for requesting inclusion
of Think Big, A Teradata Company in the powered-by NiFi page. Please let me
know if I should provide them via an email to this list also.

Thanks.
--
Jagrut

On Tue, Feb 21, 2017 at 12:06 PM, Andy LoPresto 
wrote:

> Adding Hashmap, Inc. from Chris Herrera:
>
> Begin forwarded message:
>
> *From: *Chris Herrera 
> *Subject: **Re: Powered by NiFi*
> *Date: *February 21, 2017 at 12:00:15 PM PST
> *To: *Andy LoPresto 
>
> No problem at all!
>
> Regards,
> Chris
>
> On Feb 21, 2017, 1:59 PM -0600, Andy LoPresto ,
> wrote:
>
> Thanks Chris. Mind if I forward this last message to the list in order to
> have a record of your approval?
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Feb 21, 2017, at 11:52 AM, Chris Herrera 
> wrote:
>
> Hey Andy,
>
> Sorry it took a bit to get back to you…we would love to be able to put
> something on the powered by nifi page:
>
> Company: Hashmap, Inc. (www.hashmapinc.com)
>
> Industry: Big Data / IoT
>
> Summary:
> Hashmap uses Apache NiFi to securely collect, transmit, and transform data
> for ingest and delivery into our IoT / Time Series Accelerator platform,
> allowing for outcome-based, real time analytics and visualization of oil &
> gas, utilities, manufacturing, industrial, retail, pharma, and process
> control data. Additionally, we are creating a catalog of open source,
> ready-to-run, industry specific NiFi processors and controller services for
> protocols like OPC-UA, ETP, WITSML, LAS, and many others.
>
> Regards,
> Chris
>
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Feb 16, 2017, at 2:39 PM, Andre  wrote:
>
> wow! This is looker of LookML fame. Nice one!
>
> I remember watching this video
>
> https://youtu.be/8LqnC9IjPYY
>
> about their platform a long time ago...
>
>
>
> On Fri, Feb 17, 2017 at 8:10 AM, Andy LoPresto 
> wrote:
>
>> Documentation that this was ok:
>>
>> Begin forwarded message:
>>
>> *From: *dan young 
>> *Subject: **Re: Apache NiFi + Looker*
>> *Date: *February 16, 2017 at 11:59:16 AM PST
>> *To: *Andy LoPresto 
>>
>> Sure, i think that's fine.
>>
>> On Thu, Feb 16, 2017 at 12:12 PM Andy LoPresto 
>> wrote:
>>
>>> Dan,
>>>
>>> Do I have your permission to forward the relevant part of this exchange
>>> to the users@ list in order to document that you provided this info?
>>> Thanks.
>>>
>>> Andy LoPresto
>>> alopre...@apache.org
>>> *alopresto.apa...@gmail.com *
>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>>
>>> On Feb 16, 2017, at 10:45 AM, Andy LoPresto 
>>> wrote:
>>>
>>> Got it. Thanks.
>>>
>>> Andy LoPresto
>>> alopre...@apache.org
>>> *alopresto.apa...@gmail.com *
>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>>
>>> On Feb 16, 2017, at 10:34 AM, dan young  wrote:
>>>
>>> We're a SaaS, anaylitics software...
>>>
>>> On Thu, Feb 16, 2017, 11:34 AM Andy LoPresto 
>>> wrote:
>>>
>>> This is perfect, Dan. For “industry”, would you say “Data Consulting” or
>>> do you have a better term? I’ll add this to the site. Thanks.
>>>
>>> Andy LoPresto
>>> alopre...@apache.org
>>> *alopresto.apa...@gmail.com *
>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>>
>>> On Feb 16, 2017, at 10:31 AM, dan young  wrote:
>>>
>>> Hey Andy,
>>>
>>> Got the green light to add us to the Powered by NiFi.
>>>
>>>
>>> Our website:  https://looker.com/
>>>
>>> Blurb: We're building all new data ingestion pipelines using NiFi.
>>> Existing pipeline are being migrated to NiFi as well. We have deployed NiFi
>>> clusters to ingest, transform and deliver data to various backends like
>>> Google Big Query, Amazon Redshift, and Amazon S3.
>>>
>>> LMK if you have any questions and/or need something different.
>>>
>>> Dano
>>>
>>>
>> Andy LoPresto
>> alopre...@apache.org
>> *alopresto.apa...@gmail.com *
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>
>> On Feb 16, 2017, at 11:05 AM, Andy LoPresto  wrote:
>>
>> Added Looker from Dan Young.
>>
>> Andy LoPresto
>> alopre...@apache.org
>> *alopresto.apa...@gmail.com *
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>
>> On Feb 15, 2017, at 8:58 AM, Aldrin Piri  wrote:
>>
>> Hi Giovanni,
>>
>> GoDataDriven has been included.
>>
>> Thanks!
>>
>> On Wed, Feb 15, 2017 at 11:48 AM, Giovanni Lanzani <
>> giovannilanz...@godatadriven.com> wrote:
>>
>>> Hi Joe,
>>>
>>> You can put GoDataDriven (https://godatadriven.com)
>>> Summary: GoDataDriven, a Dutch service company in the data science and
>>> engineering space, helps customers ingest and process data in real time
>>> from the most disparate devices (including but not limited to trains!).
>>>
>>> Cheers,
>>>
>>> Giovanni
>>>
>>> > -Original Message-
>>> > From: Joe Witt [mailto:

Re: Execute script and python

[BD International]

Matt,

WOW those resources are amazing thanks for the links!

Brian


On 21 Feb 2017 20:06, "Matt Burgess"  wrote:

Brian,

There is still an open Jira issue [1] to add Additional Details to the
scripting processors, which should include such best practices /
pro-tips.  In the meantime, besides my blog (thanks for the shout-out
Bryan R!), I wrote a series of articles [2] [3] [4] describing various
NiFi API methods with examples in many of the supported languages.
Also, there are tips and examples in the archives for this (the
users') and the dev email lists, shared by folks in the community.

Regards,
Matt

[1] https://issues.apache.org/jira/browse/NIFI-1954
[2] https://community.hortonworks.com/articles/75032/
executescript-cookbook-part-1.html
[3] https://community.hortonworks.com/articles/75545/
executescript-cookbook-part-2.html
[4] https://community.hortonworks.com/content/kbentry/77739/
executescript-cookbook-part-3.html


On Tue, Feb 21, 2017 at 2:57 PM, Bryan Rosander 
wrote:
> Hey Brian,
>
> One good resource around the ExecuteScript processor is Matt Burgess' blog
> [1]. Matt wrote the ExecuteScript processor and has a bunch of how-to
guides
> there.
>
> [1] http://funnifi.blogspot.com/
>
> Thanks,
> Bryan
>
> On Tue, Feb 21, 2017 at 2:51 PM, BD International
>  wrote:
>>
>> Hello,
>>
>> I am creating some python scripts which run within the ExecuteScript
>> processor. So far I've been working out what to do by looking at hello
world
>> examples like:
>>
>>
>> https://community.hortonworks.com/articles/35568/python-
script-in-nifi.html
>>
>> I've been able to do what I need to I just wanted to check if there was
>> any documentation about best practise for writing scripts or tips to best
>> use any nifi features?
>>
>> Thanks
>>
>> Brian
>
>

Re: Execute script and python

[Matt Burgess]

Brian,

There is still an open Jira issue [1] to add Additional Details to the
scripting processors, which should include such best practices /
pro-tips.  In the meantime, besides my blog (thanks for the shout-out
Bryan R!), I wrote a series of articles [2] [3] [4] describing various
NiFi API methods with examples in many of the supported languages.
Also, there are tips and examples in the archives for this (the
users') and the dev email lists, shared by folks in the community.

Regards,
Matt

[1] https://issues.apache.org/jira/browse/NIFI-1954
[2] 
https://community.hortonworks.com/articles/75032/executescript-cookbook-part-1.html
[3] 
https://community.hortonworks.com/articles/75545/executescript-cookbook-part-2.html
[4] 
https://community.hortonworks.com/content/kbentry/77739/executescript-cookbook-part-3.html


On Tue, Feb 21, 2017 at 2:57 PM, Bryan Rosander  wrote:
> Hey Brian,
>
> One good resource around the ExecuteScript processor is Matt Burgess' blog
> [1]. Matt wrote the ExecuteScript processor and has a bunch of how-to guides
> there.
>
> [1] http://funnifi.blogspot.com/
>
> Thanks,
> Bryan
>
> On Tue, Feb 21, 2017 at 2:51 PM, BD International
>  wrote:
>>
>> Hello,
>>
>> I am creating some python scripts which run within the ExecuteScript
>> processor. So far I've been working out what to do by looking at hello world
>> examples like:
>>
>>
>> https://community.hortonworks.com/articles/35568/python-script-in-nifi.html
>>
>> I've been able to do what I need to I just wanted to check if there was
>> any documentation about best practise for writing scripts or tips to best
>> use any nifi features?
>>
>> Thanks
>>
>> Brian
>
>

Re: NiFi Users: Powered by NiFi page

[Andy LoPresto]

Adding Hashmap, Inc. from Chris Herrera:

> Begin forwarded message:
> 
> From: Chris Herrera 
> Subject: Re: Powered by NiFi
> Date: February 21, 2017 at 12:00:15 PM PST
> To: Andy LoPresto 
> 
> No problem at all!
> 
> Regards,
> Chris
> 
> On Feb 21, 2017, 1:59 PM -0600, Andy LoPresto , wrote:
>> Thanks Chris. Mind if I forward this last message to the list in order to 
>> have a record of your approval?
>> 
>> Andy LoPresto
>> alopre...@apache.org 
>> alopresto.apa...@gmail.com 
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>> 
>>> On Feb 21, 2017, at 11:52 AM, Chris Herrera >> > wrote:
>>> 
>>> Hey Andy,
>>> 
>>> Sorry it took a bit to get back to you…we would love to be able to put 
>>> something on the powered by nifi page:
>>> 
>>> Company: Hashmap, Inc. (www.hashmapinc.com )
>>> 
>>> Industry: Big Data / IoT
>>> 
>>> Summary:
>>> Hashmap uses Apache NiFi to securely collect, transmit, and transform data 
>>> for ingest and delivery into our IoT / Time Series Accelerator platform, 
>>> allowing for outcome-based, real time analytics and visualization of oil & 
>>> gas, utilities, manufacturing, industrial, retail, pharma, and process 
>>> control data. Additionally, we are creating a catalog of open source, 
>>> ready-to-run, industry specific NiFi processors and controller services for 
>>> protocols like OPC-UA, ETP, WITSML, LAS, and many others.
>>> Regards,
>>> Chris

Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Feb 16, 2017, at 2:39 PM, Andre  wrote:
> 
> wow! This is looker of LookML fame. Nice one!
> 
> I remember watching this video
> 
> https://youtu.be/8LqnC9IjPYY 
> 
> about their platform a long time ago...
> 
> 
> 
> On Fri, Feb 17, 2017 at 8:10 AM, Andy LoPresto  > wrote:
> Documentation that this was ok:
> 
>> Begin forwarded message:
>> 
>> From: dan young mailto:danoyo...@gmail.com>>
>> Subject: Re: Apache NiFi + Looker
>> Date: February 16, 2017 at 11:59:16 AM PST
>> To: Andy LoPresto mailto:alopre...@apache.org>>
>> 
>> Sure, i think that's fine.
>> 
>> On Thu, Feb 16, 2017 at 12:12 PM Andy LoPresto > > wrote:
>> Dan,
>> 
>> Do I have your permission to forward the relevant part of this exchange to 
>> the users@ list in order to document that you provided this info? Thanks.
>> 
>> Andy LoPresto
>> alopre...@apache.org 
>> alopresto.apa...@gmail.com 
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>> 
>>> On Feb 16, 2017, at 10:45 AM, Andy LoPresto >> > wrote:
>>> 
>>> Got it. Thanks.
>>> 
>>> Andy LoPresto
>>> alopre...@apache.org 
>>> alopresto.apa...@gmail.com 
>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>> 
 On Feb 16, 2017, at 10:34 AM, dan young >>> > wrote:
 
 We're a SaaS, anaylitics software...
 
 On Thu, Feb 16, 2017, 11:34 AM Andy LoPresto >>> > wrote:
 This is perfect, Dan. For “industry”, would you say “Data Consulting” or 
 do you have a better term? I’ll add this to the site. Thanks.
 
 Andy LoPresto
 alopre...@apache.org 
 alopresto.apa...@gmail.com 
 PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
 
> On Feb 16, 2017, at 10:31 AM, dan young  > wrote:
> 
> Hey Andy,
> 
> Got the green light to add us to the Powered by NiFi.
> 
> 
> Our website:  https://looker.com/ 
> 
> Blurb: We're building all new data ingestion pipelines using NiFi. 
> Existing pipeline are being migrated to NiFi as well. We have deployed 
> NiFi clusters to ingest, transform and deliver data to various backends 
> like Google Big Query, Amazon Redshift, and Amazon S3.
> 
> LMK if you have any questions and/or need something different.
> 
> Dano
> 
> Andy LoPresto
> alopre...@apache.org 
> alopresto.apa...@gmail.com 
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> 
>> On Feb 16, 2017, at 11:05 AM, Andy LoPresto > > wrote:
>> 
>> Added Looker from Dan Young.
>> 
>> Andy LoPresto
>> alopre...@apache.org 
>> alopresto.apa...@gmail.com 
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>> 
>>> On Feb 15, 2017, at 8:58 AM, Aldrin Piri >> 

Execute script and python

[BD International]

Hello,

I am creating some python scripts which run within the ExecuteScript
processor. So far I've been working out what to do by looking at hello
world examples like:

https://community.hortonworks.com/articles/35568/python-script-in-nifi.html

I've been able to do what I need to I just wanted to check if there was any
documentation about best practise for writing scripts or tips to best use
any nifi features?

Thanks

Brian

Re: Execute script and python

[Bryan Rosander]

Hey Brian,

One good resource around the ExecuteScript processor is Matt Burgess' blog
[1]. Matt wrote the ExecuteScript processor and has a bunch of how-to
guides there.

[1] http://funnifi.blogspot.com/

Thanks,
Bryan

On Tue, Feb 21, 2017 at 2:51 PM, BD International <
b.deep.internatio...@gmail.com> wrote:

> Hello,
>
> I am creating some python scripts which run within the ExecuteScript
> processor. So far I've been working out what to do by looking at hello
> world examples like:
>
> https://community.hortonworks.com/articles/35568/python-
> script-in-nifi.html
>
> I've been able to do what I need to I just wanted to check if there was
> any documentation about best practise for writing scripts or tips to best
> use any nifi features?
>
> Thanks
>
> Brian
>

How to find average of two lines in NiFi?

[prabhu Mahendran]

i need to find average of two values in seperate lines.

My Csv file loo like this.,

Name,ID,Marks
Mahi,1,90
Mahi,1,90


Andy,2,100
Andy,2,100
Now i need to store that average of 2 marks in database. "Average" column
should add two marks and divide with 2 and store that result in SQL query

Table:

Name,ID,Average
Mahi,2,90
Andy,2,100
Is this possible to find average of two value's in seperate rows using NiFi?

Re: Split csv file into multipart.

[prabhu Mahendran]

Andy,

Thank you so much for your answer.

It worked.

Thanks

On Mon, Feb 20, 2017 at 9:31 PM, Andy LoPresto  wrote:

> Prabhu,
>
> I answered this question and provided a template on StackOverflow [1].
>
> [1] http://stackoverflow.com/a/42353526/70465
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Feb 20, 2017, at 6:51 AM, prabhu Mahendran 
> wrote:
>
> I have CSV File which having below contents,
>
> *Input.csv*
>
>  Sample NiFi Data demonstration for below
> Due dates 20-02-2017,23-03-2017
>
> My Input No1 inside csv,,
> Animals,Today-20.02.2017,Yesterday-19-02.2017
> Fox,21,32
> Lion,20,12
>
> My Input No2 inside csv
> Name,ID,City
> Mahi,12,UK
> And,21,US
>
> Prabh,32,LI
>
> I need to split above whole csv(Input.csv) into two parts like *InputNo1.csv* 
> and *InputNo2.csv*.
>
> For InputNo1.csv should have below contents only.,
>
> Animals,Today-20.02.2017,Yesterday-19-02.2017
> Fox,21,32
> Lion,20,12
>
> For InputNo2.csv should have below contents.,
>
> Name,ID,City
> Mahi,12,UK
> And,21,US
>
> Prabh,32,LI
>
> Is this possible to convert csv into Multiple parts in NiFi possible with
> existing processors?
>
>
>