Re: Deleting an empty group fails

[Makoto Hashimoto]

I've tested my doubt. Only deleting referenced templates in a group enables
deleting itself.

Makoto

2017-01-28 14:30 GMT+09:00 Makoto Hashimoto :

> Ｈｉ Andy,
>
> Actually, I did as you explained. There are more than 7 templates in my
> Nifi environments
> but I could delete the group after deleting  7 templates.
>
> One doubt is I have not used 7 templates in that group but I'm not sure.
>
> Makoto
>
> 2017-01-28 12:59 GMT+09:00 Andy LoPresto :
>
>> Makoto,
>>
>> For future reference, you only need to delete the templates contained in
>> the process group you are trying to delete. You can see the process group
>> ID in the templates listing — any that match the UUID of the PG you are
>> deleting need to be deleted, but others are fine.
>>
>> Andy LoPresto
>> alopre...@apache.org
>> *alopresto.apa...@gmail.com *
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>
>> On Jan 27, 2017, at 7:56 PM, Makoto Hashimoto 
>> wrote:
>>
>> Hi all,
>>
>> I could delete the process group after deleting all templates in my
>> environment.
>>
>> MH
>>
>>
>> 2017-01-28 12:38 GMT+09:00 Andy LoPresto :
>>
>>> Created NIFI-3417 [1] to address this.
>>>
>>> [1] https://issues.apache.org/jira/browse/NIFI-3417
>>>
>>> Andy LoPresto
>>> alopre...@apache.org
>>> *alopresto.apa...@gmail.com *
>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>>
>>> On Jan 27, 2017, at 7:35 PM, Joe Witt  wrote:
>>>
>>> That certainly sounds JIRA worthy though.  Templates should not be able
>>> to block the destruction of a process group.
>>>
>>> On Fri, Jan 27, 2017 at 10:32 PM, Andy LoPresto 
>>> wrote:
>>>
 Makoto,

 Templates are tied to process groups. Unfortunately, this means that to
 delete the group, you will have to delete the templates first. You can do
 this by using the “hamburger” global menu at the top right of the UI and
 selecting “Templates”, then deleting the templates which reside in this
 process group (export them to XML first if you want to save them).

 If you want to move those templates out of the PG, and you are
 comfortable working in XML, a less-enticing option is to stop the
 application, uncompress and open the flow.xml.gz file, and move the
 template entries out of the process group to root or another group, being
 sure to update any references in the XML. Then save and compress the file,
 and restart the application.

 I understand this is not an ideal solution. I’m going to file a Jira
 documenting this and suggest that the dialog allows you to select one of
 three options:

 1. Delete process group and all contained templates
 2. Delete process group and move templates to parent PG (this could
 cause issues because the components and connections will no longer be
 defined)
 3. Cancel the delete operation.

 I’ll also suggest improving the documentation to make this situation
 clearer.

 


 Andy LoPresto
 alopre...@apache.org
 *alopresto.apa...@gmail.com *
 PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

 On Jan 27, 2017, at 7:12 PM, Makoto Hashimoto 
 wrote:

 Hi,

 I'm trying to delete an empty group.

 

 However, it fails with the following error.

 

 Is something wrong with my operation and could someone would help to
 solve this problem ?

 Regards,
 MH


>>>
>>
>>
>

Re: Deleting an empty group fails

[Makoto Hashimoto]

Ｈｉ Andy,

Actually, I did as you explained. There are more than 7 templates in my
Nifi environments
but I could delete the group after deleting  7 templates.

One doubt is I have not used 7 templates in that group but I'm not sure.

Makoto

2017-01-28 12:59 GMT+09:00 Andy LoPresto :

> Makoto,
>
> For future reference, you only need to delete the templates contained in
> the process group you are trying to delete. You can see the process group
> ID in the templates listing — any that match the UUID of the PG you are
> deleting need to be deleted, but others are fine.
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Jan 27, 2017, at 7:56 PM, Makoto Hashimoto 
> wrote:
>
> Hi all,
>
> I could delete the process group after deleting all templates in my
> environment.
>
> MH
>
>
> 2017-01-28 12:38 GMT+09:00 Andy LoPresto :
>
>> Created NIFI-3417 [1] to address this.
>>
>> [1] https://issues.apache.org/jira/browse/NIFI-3417
>>
>> Andy LoPresto
>> alopre...@apache.org
>> *alopresto.apa...@gmail.com *
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>
>> On Jan 27, 2017, at 7:35 PM, Joe Witt  wrote:
>>
>> That certainly sounds JIRA worthy though.  Templates should not be able
>> to block the destruction of a process group.
>>
>> On Fri, Jan 27, 2017 at 10:32 PM, Andy LoPresto 
>> wrote:
>>
>>> Makoto,
>>>
>>> Templates are tied to process groups. Unfortunately, this means that to
>>> delete the group, you will have to delete the templates first. You can do
>>> this by using the “hamburger” global menu at the top right of the UI and
>>> selecting “Templates”, then deleting the templates which reside in this
>>> process group (export them to XML first if you want to save them).
>>>
>>> If you want to move those templates out of the PG, and you are
>>> comfortable working in XML, a less-enticing option is to stop the
>>> application, uncompress and open the flow.xml.gz file, and move the
>>> template entries out of the process group to root or another group, being
>>> sure to update any references in the XML. Then save and compress the file,
>>> and restart the application.
>>>
>>> I understand this is not an ideal solution. I’m going to file a Jira
>>> documenting this and suggest that the dialog allows you to select one of
>>> three options:
>>>
>>> 1. Delete process group and all contained templates
>>> 2. Delete process group and move templates to parent PG (this could
>>> cause issues because the components and connections will no longer be
>>> defined)
>>> 3. Cancel the delete operation.
>>>
>>> I’ll also suggest improving the documentation to make this situation
>>> clearer.
>>>
>>> 
>>>
>>>
>>> Andy LoPresto
>>> alopre...@apache.org
>>> *alopresto.apa...@gmail.com *
>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>>
>>> On Jan 27, 2017, at 7:12 PM, Makoto Hashimoto 
>>> wrote:
>>>
>>> Hi,
>>>
>>> I'm trying to delete an empty group.
>>>
>>> 
>>>
>>> However, it fails with the following error.
>>>
>>> 
>>>
>>> Is something wrong with my operation and could someone would help to
>>> solve this problem ?
>>>
>>> Regards,
>>> MH
>>>
>>>
>>
>
>

Re: Deleting an empty group fails

[Andy LoPresto]

Makoto,

For future reference, you only need to delete the templates contained in the 
process group you are trying to delete. You can see the process group ID in the 
templates listing — any that match the UUID of the PG you are deleting need to 
be deleted, but others are fine.

Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Jan 27, 2017, at 7:56 PM, Makoto Hashimoto  wrote:
> 
> Hi all,
> 
> I could delete the process group after deleting all templates in my 
> environment.
> 
> MH
> 
> 
> 2017-01-28 12:38 GMT+09:00 Andy LoPresto  >:
> Created NIFI-3417 [1] to address this.
> 
> [1] https://issues.apache.org/jira/browse/NIFI-3417 
> 
> 
> Andy LoPresto
> alopre...@apache.org 
> alopresto.apa...@gmail.com 
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> 
>> On Jan 27, 2017, at 7:35 PM, Joe Witt > > wrote:
>> 
>> That certainly sounds JIRA worthy though.  Templates should not be able to 
>> block the destruction of a process group.
>> 
>> On Fri, Jan 27, 2017 at 10:32 PM, Andy LoPresto > > wrote:
>> Makoto,
>> 
>> Templates are tied to process groups. Unfortunately, this means that to 
>> delete the group, you will have to delete the templates first. You can do 
>> this by using the “hamburger” global menu at the top right of the UI and 
>> selecting “Templates”, then deleting the templates which reside in this 
>> process group (export them to XML first if you want to save them).
>> 
>> If you want to move those templates out of the PG, and you are comfortable 
>> working in XML, a less-enticing option is to stop the application, 
>> uncompress and open the flow.xml.gz file, and move the template entries out 
>> of the process group to root or another group, being sure to update any 
>> references in the XML. Then save and compress the file, and restart the 
>> application.
>> 
>> I understand this is not an ideal solution. I’m going to file a Jira 
>> documenting this and suggest that the dialog allows you to select one of 
>> three options:
>> 
>> 1. Delete process group and all contained templates
>> 2. Delete process group and move templates to parent PG (this could cause 
>> issues because the components and connections will no longer be defined)
>> 3. Cancel the delete operation.
>> 
>> I’ll also suggest improving the documentation to make this situation clearer.
>> 
>> 
>> 
>> 
>> Andy LoPresto
>> alopre...@apache.org 
>> alopresto.apa...@gmail.com 
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>> 
>>> On Jan 27, 2017, at 7:12 PM, Makoto Hashimoto >> > wrote:
>>> 
>>> Hi,
>>> 
>>> I'm trying to delete an empty group.
>>> 
>>> 
>>> 
>>> However, it fails with the following error.
>>> 
>>> 
>>> 
>>> Is something wrong with my operation and could someone would help to
>>> solve this problem ?
>>> 
>>> Regards,
>>> MH
> 
> 



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Deleting an empty group fails

[Makoto Hashimoto]

Hi all,

I could delete the process group after deleting all templates in my
environment.

MH


2017-01-28 12:38 GMT+09:00 Andy LoPresto :

> Created NIFI-3417 [1] to address this.
>
> [1] https://issues.apache.org/jira/browse/NIFI-3417
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Jan 27, 2017, at 7:35 PM, Joe Witt  wrote:
>
> That certainly sounds JIRA worthy though.  Templates should not be able to
> block the destruction of a process group.
>
> On Fri, Jan 27, 2017 at 10:32 PM, Andy LoPresto 
> wrote:
>
>> Makoto,
>>
>> Templates are tied to process groups. Unfortunately, this means that to
>> delete the group, you will have to delete the templates first. You can do
>> this by using the “hamburger” global menu at the top right of the UI and
>> selecting “Templates”, then deleting the templates which reside in this
>> process group (export them to XML first if you want to save them).
>>
>> If you want to move those templates out of the PG, and you are
>> comfortable working in XML, a less-enticing option is to stop the
>> application, uncompress and open the flow.xml.gz file, and move the
>> template entries out of the process group to root or another group, being
>> sure to update any references in the XML. Then save and compress the file,
>> and restart the application.
>>
>> I understand this is not an ideal solution. I’m going to file a Jira
>> documenting this and suggest that the dialog allows you to select one of
>> three options:
>>
>> 1. Delete process group and all contained templates
>> 2. Delete process group and move templates to parent PG (this could cause
>> issues because the components and connections will no longer be defined)
>> 3. Cancel the delete operation.
>>
>> I’ll also suggest improving the documentation to make this situation
>> clearer.
>>
>> 
>>
>>
>> Andy LoPresto
>> alopre...@apache.org
>> *alopresto.apa...@gmail.com *
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>
>> On Jan 27, 2017, at 7:12 PM, Makoto Hashimoto 
>> wrote:
>>
>> Hi,
>>
>> I'm trying to delete an empty group.
>>
>> 
>>
>> However, it fails with the following error.
>>
>> 
>>
>> Is something wrong with my operation and could someone would help to
>> solve this problem ?
>>
>> Regards,
>> MH
>>
>>
>

Re: Deleting an empty group fails

[Andy LoPresto]

Created NIFI-3417 [1] to address this.

[1] https://issues.apache.org/jira/browse/NIFI-3417

Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Jan 27, 2017, at 7:35 PM, Joe Witt  wrote:
> 
> That certainly sounds JIRA worthy though.  Templates should not be able to 
> block the destruction of a process group.
> 
> On Fri, Jan 27, 2017 at 10:32 PM, Andy LoPresto  > wrote:
> Makoto,
> 
> Templates are tied to process groups. Unfortunately, this means that to 
> delete the group, you will have to delete the templates first. You can do 
> this by using the “hamburger” global menu at the top right of the UI and 
> selecting “Templates”, then deleting the templates which reside in this 
> process group (export them to XML first if you want to save them).
> 
> If you want to move those templates out of the PG, and you are comfortable 
> working in XML, a less-enticing option is to stop the application, uncompress 
> and open the flow.xml.gz file, and move the template entries out of the 
> process group to root or another group, being sure to update any references 
> in the XML. Then save and compress the file, and restart the application.
> 
> I understand this is not an ideal solution. I’m going to file a Jira 
> documenting this and suggest that the dialog allows you to select one of 
> three options:
> 
> 1. Delete process group and all contained templates
> 2. Delete process group and move templates to parent PG (this could cause 
> issues because the components and connections will no longer be defined)
> 3. Cancel the delete operation.
> 
> I’ll also suggest improving the documentation to make this situation clearer.
> 
> 
> 
> 
> Andy LoPresto
> alopre...@apache.org 
> alopresto.apa...@gmail.com 
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> 
>> On Jan 27, 2017, at 7:12 PM, Makoto Hashimoto > > wrote:
>> 
>> Hi,
>> 
>> I'm trying to delete an empty group.
>> 
>> 
>> 
>> However, it fails with the following error.
>> 
>> 
>> 
>> Is something wrong with my operation and could someone would help to
>> solve this problem ?
>> 
>> Regards,
>> MH



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Deleting an empty group fails

[Joe Witt]

That certainly sounds JIRA worthy though.  Templates should not be able to
block the destruction of a process group.

On Fri, Jan 27, 2017 at 10:32 PM, Andy LoPresto 
wrote:

> Makoto,
>
> Templates are tied to process groups. Unfortunately, this means that to
> delete the group, you will have to delete the templates first. You can do
> this by using the “hamburger” global menu at the top right of the UI and
> selecting “Templates”, then deleting the templates which reside in this
> process group (export them to XML first if you want to save them).
>
> If you want to move those templates out of the PG, and you are comfortable
> working in XML, a less-enticing option is to stop the application,
> uncompress and open the flow.xml.gz file, and move the template entries out
> of the process group to root or another group, being sure to update any
> references in the XML. Then save and compress the file, and restart the
> application.
>
> I understand this is not an ideal solution. I’m going to file a Jira
> documenting this and suggest that the dialog allows you to select one of
> three options:
>
> 1. Delete process group and all contained templates
> 2. Delete process group and move templates to parent PG (this could cause
> issues because the components and connections will no longer be defined)
> 3. Cancel the delete operation.
>
> I’ll also suggest improving the documentation to make this situation
> clearer.
>
>
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Jan 27, 2017, at 7:12 PM, Makoto Hashimoto 
> wrote:
>
> Hi,
>
> I'm trying to delete an empty group.
>
> 
>
> However, it fails with the following error.
>
> 
>
> Is something wrong with my operation and could someone would help to
> solve this problem ?
>
> Regards,
> MH
>
>
>

Deleting an empty group fails

[Makoto Hashimoto]

Hi,

I'm trying to delete an empty group.

[image: 埋め込み画像 1]

However, it fails with the following error.

[image: 埋め込み画像 2]

Is something wrong with my operation and could someone would help to
solve this problem ?

Regards,
MH

Re: Custom processors in MiNiFi

[Andy LoPresto]

One other note; you may find additional help on our developers list - 
d...@nifi.apache.org. This list is more focused on user issues and 
functionality, while that list gets much deeper into the weeds on coding.

Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Jan 27, 2017, at 11:04 AM, Andy LoPresto  wrote:
> 
> Hi Aakash.
> 
> Last summer I had an intern working for me who investigated using machine 
> learning (unsupervised anomaly detection using kNN and LOF) against NiFi 
> provenance data to perform error identification and build a processor 
> recommendation engine. I can’t share the work as it is company internal, but 
> there is definitely a growing community and interest in what you’re 
> discussing.
> 
> If you truly want to distribute the computational load of performing the 
> analysis to edge nodes, writing custom processors is likely a requirement. 
> Can I make two suggestions before you begin writing code, though? First, 
> investigate if you could deploy something like scikit-learn (Python) [1] or 
> Apache Spark-ML [2] to reside alongside NiFi on the edge nodes (obviously 
> depends on HW resources). Our early efforts involved writing custom NiFi 
> code, but it turned out it was much easier to offload the data to 
> scikit-learn and then ingest the results back into NiFi to continue data 
> flow, while leaving the computation to an external system.
> 
> If you really want the computation to be running inside the NiFi JVM, also 
> look at the ExecuteScript processor before trying to write a custom 
> processor. While NiFi makes it easy to deploy custom code, the SDLC can 
> provide a few constant delays — after you generate the Maven pom for the NAR, 
> you will have to write the code in an IDE, test it, compile, build the NAR, 
> drop it into the NiFi lib, and restart the entire application every time you 
> make a change. To prototype your model, I recommend using the ES processor, 
> which will provide immediate feedback. It also abstracts a lot of the 
> boilerplate framework so you can hyper focus on the domain work. Matt Burgess 
> has written a number of great articles which should get you up and running 
> with it [3].
> 
> Once you have a model and computation you’re confident in, then it’s easy to 
> translate it to a dedicated custom processor and deploy it. I find this 
> methodology saves me a lot of time and a bit of frustration. Good luck. I’m 
> very curious to see what your work yields.
> 
> [1] http://scikit-learn.org/stable/ 
> [2] https://spark.apache.org/mllib/ 
> [3] https://funnifi.blogspot.com 
> 
> 
> 
> Andy LoPresto
> alopre...@apache.org 
> alopresto.apa...@gmail.com 
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> 
>> On Jan 27, 2017, at 5:45 AM, Aldrin Piri > > wrote:
>> 
>> Hi, Aakash!
>> 
>> To my knowledge, I have not seen any discussion about such processors on the 
>> lists specifically although have heard people mentioning assorted libraries 
>> that might be a good fit for the NiFi ecosystem's intended purposes.  There 
>> has been some foundational work such as the following issues which allow 
>> processors to make use of the state management features in NiFi for the sake 
>> of managing the flow of data to do some higher level inspection/analysis.
>> 
>> https://issues.apache.org/jira/browse/NIFI-1582 
>> 
>> https://issues.apache.org/jira/browse/NIFI-1682 
>> 
>> https://issues.apache.org/jira/browse/NIFI-2590 
>> 
>> 
>> If my understanding of your question is correct, I believe your notion of 
>> distribution may not directly align with the intended focus of NiFi, but 
>> certainly could be some aspects that work.  Would you be willing to expand 
>> in greater detail how you would envision such processors interacting with 
>> data and possibly provide some of the libraries you were considering in your 
>> initial message?
>> 
>> Thanks!
>> 
>> --aldrin
>> 
>> On Fri, Jan 27, 2017 at 7:38 AM, Aakash Khochare 
>> > 
>> wrote:
>> Greetings,
>> 
>> While I understand that the primary use of NiFi/MiNiFi is for secure data 
>> ingress with the added benefit of Provenance, what are the views of the 
>> community on writing Processors that implement Machine Learning Algorithms 
>> and distribute them across Edge+ Cloud using NiFi and MiNiFi? Has anyone 
>> tried writing such processors?
>> 
>> Regards,
>> 
>> Aakash Khochare
>> 
>> 
>> 
> 



signature.asc
Description: Message signed with 

Re: Custom processors in MiNiFi

[Andy LoPresto]

Hi Aakash.

Last summer I had an intern working for me who investigated using machine 
learning (unsupervised anomaly detection using kNN and LOF) against NiFi 
provenance data to perform error identification and build a processor 
recommendation engine. I can’t share the work as it is company internal, but 
there is definitely a growing community and interest in what you’re discussing.

If you truly want to distribute the computational load of performing the 
analysis to edge nodes, writing custom processors is likely a requirement. Can 
I make two suggestions before you begin writing code, though? First, 
investigate if you could deploy something like scikit-learn (Python) [1] or 
Apache Spark-ML [2] to reside alongside NiFi on the edge nodes (obviously 
depends on HW resources). Our early efforts involved writing custom NiFi code, 
but it turned out it was much easier to offload the data to scikit-learn and 
then ingest the results back into NiFi to continue data flow, while leaving the 
computation to an external system.

If you really want the computation to be running inside the NiFi JVM, also look 
at the ExecuteScript processor before trying to write a custom processor. While 
NiFi makes it easy to deploy custom code, the SDLC can provide a few constant 
delays — after you generate the Maven pom for the NAR, you will have to write 
the code in an IDE, test it, compile, build the NAR, drop it into the NiFi lib, 
and restart the entire application every time you make a change. To prototype 
your model, I recommend using the ES processor, which will provide immediate 
feedback. It also abstracts a lot of the boilerplate framework so you can hyper 
focus on the domain work. Matt Burgess has written a number of great articles 
which should get you up and running with it [3].

Once you have a model and computation you’re confident in, then it’s easy to 
translate it to a dedicated custom processor and deploy it. I find this 
methodology saves me a lot of time and a bit of frustration. Good luck. I’m 
very curious to see what your work yields.

[1] http://scikit-learn.org/stable/ 
[2] https://spark.apache.org/mllib/ 
[3] https://funnifi.blogspot.com



Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Jan 27, 2017, at 5:45 AM, Aldrin Piri  wrote:
> 
> Hi, Aakash!
> 
> To my knowledge, I have not seen any discussion about such processors on the 
> lists specifically although have heard people mentioning assorted libraries 
> that might be a good fit for the NiFi ecosystem's intended purposes.  There 
> has been some foundational work such as the following issues which allow 
> processors to make use of the state management features in NiFi for the sake 
> of managing the flow of data to do some higher level inspection/analysis.
> 
> https://issues.apache.org/jira/browse/NIFI-1582 
> 
> https://issues.apache.org/jira/browse/NIFI-1682 
> 
> https://issues.apache.org/jira/browse/NIFI-2590 
> 
> 
> If my understanding of your question is correct, I believe your notion of 
> distribution may not directly align with the intended focus of NiFi, but 
> certainly could be some aspects that work.  Would you be willing to expand in 
> greater detail how you would envision such processors interacting with data 
> and possibly provide some of the libraries you were considering in your 
> initial message?
> 
> Thanks!
> 
> --aldrin
> 
> On Fri, Jan 27, 2017 at 7:38 AM, Aakash Khochare 
> > 
> wrote:
> Greetings,
> 
> While I understand that the primary use of NiFi/MiNiFi is for secure data 
> ingress with the added benefit of Provenance, what are the views of the 
> community on writing Processors that implement Machine Learning Algorithms 
> and distribute them across Edge+ Cloud using NiFi and MiNiFi? Has anyone 
> tried writing such processors?
> 
> Regards,
> 
> Aakash Khochare
> 
> 
> 



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: nifi at AWS

[mohammed shambakey]

Thank you very much. I think setting the "nifi.remote.input.host" to the
public DNS name did the trick.

BTW, is there a difference in the security group between "0.0.0.0/0" and
"::/0"?

Regards

On Fri, Jan 27, 2017 at 1:05 AM, Koji Kawamura 
wrote:

> Hi Mohammed,
>
> For users who are trying to connect between localhost and AWS, I wrote
> a post on "How to send file from localhost to AWS EC2 using NiFi
> Site-to-Site?"
> http://ijokarumawak.github.io/nifi/2017/01/27/nifi-s2s-local-to-aws/
>
> There must be some configuration or environmental difference, hope we
> can find it soon.
>
> Thanks,
> Koji
>
> On Thu, Jan 26, 2017 at 8:53 PM, mohammed shambakey
>  wrote:
> > Thank you all for your help
> >
> > I'm using nifi 1.1.1 and the HTTP protocol. I can access the remote
> instance
> > in AWS in my local browser, but I'm trying to upload a local file to this
> > remote instance (using GetFile processor), then it fails to make the
> > transaction.
> >
> > Regards
> >
> > On Wed, Jan 25, 2017 at 9:51 PM, Koji Kawamura 
> > wrote:
> >>
> >> Hi Mohammed,
> >>
> >> Which version of NiFi are you using? If it's 1.0.0 or later, you can
> >> choose 'HTTP' as 'Transport Protocol' in RemoteProcessGroup
> >> configuration in your local NiFi, this is what Andrew suggested
> >> earlier.
> >>
> >> With HTTP transport protocol, the local NiFi will use HTTP port (8080
> >> in your case) to send flow files to that remote NiFi running on AWS.
> >> If you can access remote NiFi's UI from browser without issue, this
> >> should work.
> >>
> >> As a side note, if you prefer using RAW transport protocol, then you'd
> >> have to open additional port, which is defined as
> >> nifi.remote.input.socket.port, in AWS security group setting. Since
> >> HTTP transport protocol doesn't require this, HTTP is more advisable.
> >>
> >> Thanks,
> >> Koji
> >>
> >> On Wed, Jan 25, 2017 at 1:07 AM, mohammed shambakey
> >>  wrote:
> >> > yes
> >> >
> >> > On Tue, Jan 24, 2017 at 10:40 AM, Antunes, Ravel
> >> > 
> >> > wrote:
> >> >>
> >> >> Have you set nifi.remote.input.host to the EC2 instance public DNS?
> >> >>
> >> >>
> >> >>
> >> >> From: mohammed shambakey 
> >> >> Reply-To: "users@nifi.apache.org" 
> >> >> Date: Tuesday, January 24, 2017 at 9:25 AM
> >> >> To: "users@nifi.apache.org" 
> >> >> Subject: Re: nifi at AWS
> >> >>
> >> >>
> >> >>
> >> >> Thank you all.
> >> >>
> >> >>
> >> >>
> >> >> I'm already using HTTP, and "transmission" and "ports" are open (I
> >> >> think
> >> >> this is what you mean by nifi.remote.input.socket.port, right?), but
> >> >> still
> >> >> the same problem.
> >> >>
> >> >>
> >> >>
> >> >> I can access the remote instance from another AWS instance in the
> same
> >> >> VPC
> >> >> (I didn't try another VPC), but from my local machine to the remote
> AWS
> >> >> instance, transaction fails.
> >> >>
> >> >>
> >> >>
> >> >> I tried to open all input TCP tranffic to the AWS instance, but AWS
> >> >> didn't
> >> >> allow that. If it can't be solved, I think I'll just use same EC2
> >> >> instances
> >> >> running in the same VPC.
> >> >>
> >> >>
> >> >>
> >> >> Regards
> >> >>
> >> >>
> >> >>
> >> >> On Sun, Jan 22, 2017 at 4:04 PM, Andrew Grande 
> >> >> wrote:
> >> >>
> >> >> Isn't it more advisable to use the HTTP mode instead, i.e. no
> >> >> additional
> >> >> ports to open? Make sure to change the client RPG mode to http from
> RAW
> >> >> (in
> >> >> the UI).
> >> >>
> >> >> Andrew
> >> >>
> >> >>
> >> >>
> >> >> On Sun, Jan 22, 2017, 10:47 AM Bryan Bende  wrote:
> >> >>
> >> >> Hello,
> >> >>
> >> >>
> >> >>
> >> >> I'm assuming you are using site-to-site since you mentioned failing
> to
> >> >> create a transaction.
> >> >>
> >> >>
> >> >>
> >> >> In nifi.properties on the AWS instance, there is probably a value for
> >> >> nifi.remote.input.socket.port which would also need to be opened.
> >> >>
> >> >>
> >> >>
> >> >> -Bryan
> >> >>
> >> >>
> >> >>
> >> >> On Sat, Jan 21, 2017 at 7:00 PM, mohammed shambakey
> >> >> 
> >> >> wrote:
> >> >>
> >> >> Hi
> >> >>
> >> >>
> >> >>
> >> >> I'm trying to send a file from a local nifi instatnce to a remote
> nifi
> >> >> instance in AWS. Security rules at remote instance has port 8080
> >> >> opened, yet
> >> >> each time I try to send the file, local nifi says it failed to create
> >> >> transaction to the remote instance.
> >> >>
> >> >>
> >> >>
> >> >> Regards
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >>
> >> >> Mohammed
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >>
> >> >> Mohammed
> >> >
> >> >
> >> >
> >> >
> >> > --
> >> > Mohammed
> >
> >
> >
> >
> > --
> > Mohammed
>



-- 
Mohammed

Re: NiFi 1.1.0 stuck starting, no errors

[Joe Witt]

...ok so we definitely want to figure out why there was another nifi
process still lingering around in the background.  Do you have any
idea why that might have been? Perhaps the startup script has some
corner cases not well covered just yet.

On Fri, Jan 27, 2017 at 12:39 PM, Peter Wicks (pwicks)
 wrote:
> Restarting the box fixed it.
>
>
>
> From: Bryan Rosander [mailto:brosan...@apache.org]
> Sent: Friday, January 27, 2017 8:51 AM
> To: users@nifi.apache.org
> Subject: Re: NiFi 1.1.0 stuck starting, no errors
>
>
>
> Hey Peter,
>
>
>
> The relevant section of the thread dump is below.  It looks like startup is
> blocking while trying to get a file lock for the flowfile repository.  Are
> you positive that there is only one instance of NiFi running with the
> repository directory you've configured?  If so, maybe something failed to
> release the lock in which case a reboot might be worth a try.
>
>
>
> Thanks,
>
> Bryan
>
>
>
> "main" #1 prio=5 os_prio=0 tid=0x7f8da000d000 nid=0xcd68 runnable
> [0x7f8da6daa000]
>
>java.lang.Thread.State: RUNNABLE
>
> at sun.nio.ch.FileDispatcherImpl.lock0(Native Method)
>
> at sun.nio.ch.FileDispatcherImpl.lock(FileDispatcherImpl.java:90)
>
> at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1073)
>
> at java.nio.channels.FileChannel.lock(FileChannel.java:1053)
>
> at
> org.wali.MinimalLockingWriteAheadLog.(MinimalLockingWriteAheadLog.java:187)
>
> at
> org.wali.MinimalLockingWriteAheadLog.(MinimalLockingWriteAheadLog.java:112)
>
> at
> org.apache.nifi.controller.repository.WriteAheadFlowFileRepository.initialize(WriteAheadFlowFileRepository.java:148)
>
> at
> org.apache.nifi.controller.FlowController.createFlowFileRepository(FlowController.java:638)
>
> - locked <0x0006bfc17cc0> (a
> org.apache.nifi.controller.repository.WriteAheadFlowFileRepository)
>
> at
> org.apache.nifi.controller.FlowController.(FlowController.java:447)
>
> at
> org.apache.nifi.controller.FlowController.createStandaloneInstance(FlowController.java:375)
>
> at
> org.apache.nifi.spring.FlowControllerFactoryBean.getObject(FlowControllerFactoryBean.java:74)
>
> at
> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
>
> at
> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
>
> - locked <0x00067a34fbf8> (a
> java.util.concurrent.ConcurrentHashMap)
>
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1585)
>
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:317)
>
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
>
> at
> org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1060)
>
> at
> org.apache.nifi.spring.StandardFlowServiceFactoryBean.getObject(StandardFlowServiceFactoryBean.java:48)
>
> at
> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
>
> at
> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
>
> - locked <0x00067a34fbf8> (a
> java.util.concurrent.ConcurrentHashMap)
>
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1585)
>
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:317)
>
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
>
> at
> org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1060)
>
> at
> org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:53)
>
> at
> org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:837)
>
> at
> org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:533)
>
> at
> org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:810)
>
> at
> org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:345)
>
> at
> org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1404)
>
> at
> org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1366)
>
> at
> 

Re: NiFi 1.1.0 stuck starting, no errors

[Bryan Rosander]

Hey Peter,

Weighing in at 200MB means it is likely a heap dump (all working memory).
A thread dump should be much smaller.  If using jcmd, please try
Thread.print as the command.  Another alternative would be to use jstack
[PID].

Thanks,
Bryan

On Fri, Jan 27, 2017 at 9:24 AM, Peter Wicks (pwicks) 
wrote:

> It clocked in at around 200MB’s… https://drive.google.com/file/d/
> 0B4yjbe5sOeAuRXAxT3Z6anRRN1k/view?usp=sharing
>
>
>
>
>
> *From:* Andy LoPresto [mailto:alopre...@apache.org]
> *Sent:* Thursday, January 26, 2017 3:06 PM
> *To:* users@nifi.apache.org
> *Subject:* Re: NiFi 1.1.0 stuck starting, no errors
>
>
>
> Hi Peter,
>
> Can you provide a thread dump of the process? You should be able to do
> this via the jcmd tool [1].
>
> [1] https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/
> tooldescr006.html#BABEHABG  javase/8/docs/technotes/guides/troubleshoot/tooldescr006.html#BABEHABG>
>
> Andy LoPresto
> alopre...@apache.org
> alopresto.apa...@gmail.com
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> > On Jan 26, 2017, at 12:42 PM, Peter Wicks (pwicks) 
> wrote:
> >
> > I’m looking for help in troubleshooting my NiFi 1.1.0 install.  It’s
> been running stably for some time, but I restarted it this morning when I
> deployed an updated custom NAR. Now it gets stuck at startup, see logs at
> the end.
>
> > There are no error messages, and the processes don’t die. The process
> just seems to be hanging waiting for something.
> >
> > · My first thought was to try rolling back the modified nar, and
> even just removing the nar all together since it was custom.  Neither of
> these made any difference.
>
> > · I also tried deleting the “work” folder, which has fixed nar
> versioning issues for me in the past (not really related, but was worth a
> shot). This made no difference.
>
> > · NiFi is set to start with java.arg.2=-Xms4G and
> java.arg.3=-Xmx8G, 22GB’s of free RAM are available on the system (out of
> some 60GB’s total).
>
> > · I’ve checked running processes, and when I stop NiFi no rouge
> instances are left running.
> > · Since NiFi gets stuck right around the JettyServer step I
> checked to see if any processes were using port 8443. No other processes
> are using this port.
>
> > · I thought maybe a key file was being locked, but with NiFi off
> `lsof | grep nifi` returns no locked files.
> >
> > Nifi-app Log:
> > 2017-01-26 20:23:43,359 INFO [main] org.eclipse.jetty.util.log Logging
> initialized @90357ms
> > 2017-01-26 20:23:43,418 INFO [main] org.apache.nifi.web.server.JettyServer
> Configuring Jetty for HTTPs on port: 8443
> > 2017-01-26 20:23:43,691 INFO [main] org.apache.nifi.web.server.JettyServer
> Loading WAR: /data/nifi/nifi-1.1.0/./work/nar/extensions/nifi-media-nar-
> 1.1.0.nar-unpacked/META-INF/bundled-dependencies/nifi-image-viewer-1.1.0.war
> with context path set to /nifi-image-viewer-1.1.0
>
> > 2017-01-26 20:23:43,702 INFO [main] org.apache.nifi.web.server.JettyServer
> Loading WAR: /data/nifi/nifi-1.1.0/./work/nar/extensions/nifi-update-
> attribute-nar-1.1.0.nar-unpacked/META-INF/bundled-
> dependencies/nifi-update-attribute-ui-1.1.0.war with context path set to
> /nifi-update-attribute-ui-1.1.0
>
> > 2017-01-26 20:23:43,703 INFO [main] org.apache.nifi.web.server.JettyServer
> Loading UI extension [ProcessorConfiguration, /nifi-update-attribute-ui-1.1.0]
> for [org.apache.nifi.processors.attributes.UpdateAttribute]
>
> > 2017-01-26 20:23:43,713 INFO [main] org.apache.nifi.web.server.JettyServer
> Loading WAR: /data/nifi/nifi-1.1.0/./work/nar/extensions/nifi-standard-
> nar-1.1.0.nar-unpacked/META-INF/bundled-dependencies/nifi-
> standard-content-viewer-1.1.0.war with context path set to
> /nifi-standard-content-viewer-1.1.0
>
> > 2017-01-26 20:23:43,723 INFO [main] org.apache.nifi.web.server.JettyServer
> Loading WAR: /data/nifi/nifi-1.1.0/./work/nar/extensions/nifi-standard-
> nar-1.1.0.nar-unpacked/META-INF/bundled-dependencies/nifi-
> jolt-transform-json-ui-1.1.0.war with context path set to
> /nifi-jolt-transform-json-ui-1.1.0
>
> > 2017-01-26 20:23:43,724 INFO [main] org.apache.nifi.web.server.JettyServer
> Loading UI extension [ProcessorConfiguration, 
> /nifi-jolt-transform-json-ui-1.1.0]
> for [org.apache.nifi.processors.standard.JoltTransformJSON]
>
> > 2017-01-26 20:23:43,729 INFO [main] org.apache.nifi.web.server.JettyServer
> Loading WAR: /data/nifi/nifi-1.1.0/./work/nar/framework/nifi-framework-
> nar-1.1.0.nar-unpacked/META-INF/bundled-dependencies/nifi-web-ui-1.1.0.war
> with context path set to /nifi
>
> > 2017-01-26 20:23:43,733 INFO [main] org.apache.nifi.web.server.JettyServer
> Loading WAR: /data/nifi/nifi-1.1.0/./work/nar/framework/nifi-framework-
> nar-1.1.0.nar-unpacked/META-INF/bundled-dependencies/nifi-web-api-1.1.0.war
> with context path set to /nifi-api
>
> > 2017-01-26 20:23:43,735 INFO [main] 

Re: ExecuteSQL with LIMIT

[Mike Harding]

Please ignore this - I had the processor on the wrong scheduler causing the
behaviour I was seeing.

Mike

On 27 January 2017 at 13:00, Mike Harding  wrote:

> Hi Mike,
>
> I want to run a select command with ExecuteSQL but it doesn't seem to work
> and the query pulls all of the records. Is it supported or will another
> processor type allow me to LIMIT the result set returned?
>
> Thanks,
> Mike
>

RE: NiFi 1.1.0 stuck starting, no errors

[Peter Wicks (pwicks)]

It clocked in at around 200MB’s… 
https://drive.google.com/file/d/0B4yjbe5sOeAuRXAxT3Z6anRRN1k/view?usp=sharing


From: Andy LoPresto [mailto:alopre...@apache.org]
Sent: Thursday, January 26, 2017 3:06 PM
To: users@nifi.apache.org
Subject: Re: NiFi 1.1.0 stuck starting, no errors


Hi Peter,

Can you provide a thread dump of the process? You should be able to do this via 
the jcmd tool [1].

[1] 
https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr006.html#BABEHABG
 


Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Jan 26, 2017, at 12:42 PM, Peter Wicks (pwicks) 
> > wrote:
>
> I’m looking for help in troubleshooting my NiFi 1.1.0 install.  It’s been 
> running stably for some time, but I restarted it this morning when I deployed 
> an updated custom NAR. Now it gets stuck at startup, see logs at the end.

> There are no error messages, and the processes don’t die. The process just 
> seems to be hanging waiting for something.
>
> · My first thought was to try rolling back the modified nar, and even 
> just removing the nar all together since it was custom.  Neither of these 
> made any difference.

> · I also tried deleting the “work” folder, which has fixed nar 
> versioning issues for me in the past (not really related, but was worth a 
> shot). This made no difference.

> · NiFi is set to start with java.arg.2=-Xms4G and java.arg.3=-Xmx8G, 
> 22GB’s of free RAM are available on the system (out of some 60GB’s total).

> · I’ve checked running processes, and when I stop NiFi no rouge 
> instances are left running.
> · Since NiFi gets stuck right around the JettyServer step I checked 
> to see if any processes were using port 8443. No other processes are using 
> this port.

> · I thought maybe a key file was being locked, but with NiFi off 
> `lsof | grep nifi` returns no locked files.
>
> Nifi-app Log:
> 2017-01-26 20:23:43,359 INFO [main] org.eclipse.jetty.util.log Logging 
> initialized @90357ms
> 2017-01-26 20:23:43,418 INFO [main] org.apache.nifi.web.server.JettyServer 
> Configuring Jetty for HTTPs on port: 8443
> 2017-01-26 20:23:43,691 INFO [main] org.apache.nifi.web.server.JettyServer 
> Loading WAR: 
> /data/nifi/nifi-1.1.0/./work/nar/extensions/nifi-media-nar-1.1.0.nar-unpacked/META-INF/bundled-dependencies/nifi-image-viewer-1.1.0.war
>  with context path set to /nifi-image-viewer-1.1.0

> 2017-01-26 20:23:43,702 INFO [main] org.apache.nifi.web.server.JettyServer 
> Loading WAR: 
> /data/nifi/nifi-1.1.0/./work/nar/extensions/nifi-update-attribute-nar-1.1.0.nar-unpacked/META-INF/bundled-dependencies/nifi-update-attribute-ui-1.1.0.war
>  with context path set to /nifi-update-attribute-ui-1.1.0

> 2017-01-26 20:23:43,703 INFO [main] org.apache.nifi.web.server.JettyServer 
> Loading UI extension [ProcessorConfiguration, 
> /nifi-update-attribute-ui-1.1.0] for 
> [org.apache.nifi.processors.attributes.UpdateAttribute]

> 2017-01-26 20:23:43,713 INFO [main] org.apache.nifi.web.server.JettyServer 
> Loading WAR: 
> /data/nifi/nifi-1.1.0/./work/nar/extensions/nifi-standard-nar-1.1.0.nar-unpacked/META-INF/bundled-dependencies/nifi-standard-content-viewer-1.1.0.war
>  with context path set to /nifi-standard-content-viewer-1.1.0

> 2017-01-26 20:23:43,723 INFO [main] org.apache.nifi.web.server.JettyServer 
> Loading WAR: 
> /data/nifi/nifi-1.1.0/./work/nar/extensions/nifi-standard-nar-1.1.0.nar-unpacked/META-INF/bundled-dependencies/nifi-jolt-transform-json-ui-1.1.0.war
>  with context path set to /nifi-jolt-transform-json-ui-1.1.0

> 2017-01-26 20:23:43,724 INFO [main] org.apache.nifi.web.server.JettyServer 
> Loading UI extension [ProcessorConfiguration, 
> /nifi-jolt-transform-json-ui-1.1.0] for 
> [org.apache.nifi.processors.standard.JoltTransformJSON]

> 2017-01-26 20:23:43,729 INFO [main] org.apache.nifi.web.server.JettyServer 
> Loading WAR: 
> /data/nifi/nifi-1.1.0/./work/nar/framework/nifi-framework-nar-1.1.0.nar-unpacked/META-INF/bundled-dependencies/nifi-web-ui-1.1.0.war
>  with context path set to /nifi

> 2017-01-26 20:23:43,733 INFO [main] org.apache.nifi.web.server.JettyServer 
> Loading WAR: 
> /data/nifi/nifi-1.1.0/./work/nar/framework/nifi-framework-nar-1.1.0.nar-unpacked/META-INF/bundled-dependencies/nifi-web-api-1.1.0.war
>  with context path set to /nifi-api

> 2017-01-26 20:23:43,735 INFO [main] org.apache.nifi.web.server.JettyServer 
> Loading WAR: 
> /data/nifi/nifi-1.1.0/./work/nar/framework/nifi-framework-nar-1.1.0.nar-unpacked/META-INF/bundled-dependencies/nifi-web-content-viewer-1.1.0.war
>  with context path set to /nifi-content-viewer

> 2017-01-26 20:23:43,738 INFO [main] org.apache.nifi.web.server.JettyServer 
> 

Re: Custom processors in MiNiFi

[Aldrin Piri]

Hi, Aakash!

To my knowledge, I have not seen any discussion about such processors on
the lists specifically although have heard people mentioning assorted
libraries that might be a good fit for the NiFi ecosystem's intended
purposes.  There has been some foundational work such as the following
issues which allow processors to make use of the state management features
in NiFi for the sake of managing the flow of data to do some higher level
inspection/analysis.

https://issues.apache.org/jira/browse/NIFI-1582
https://issues.apache.org/jira/browse/NIFI-1682
https://issues.apache.org/jira/browse/NIFI-2590

If my understanding of your question is correct, I believe your notion of
distribution may not directly align with the intended focus of NiFi, but
certainly could be some aspects that work.  Would you be willing to expand
in greater detail how you would envision such processors interacting with
data and possibly provide some of the libraries you were considering in
your initial message?

Thanks!

--aldrin

On Fri, Jan 27, 2017 at 7:38 AM, Aakash Khochare <
aakhoch...@grads.cds.iisc.ac.in> wrote:

> Greetings,
>
> While I understand that the primary use of NiFi/MiNiFi is for secure data
> ingress with the added benefit of Provenance, what are the views of the
> community on writing Processors that implement Machine Learning Algorithms
> and distribute them across Edge+ Cloud using NiFi and MiNiFi? Has anyone
> tried writing such processors?
>
>
> Regards,
>
> Aakash Khochare
>
>
>

ExecuteSQL with LIMIT

[Mike Harding]

Hi Mike,

I want to run a select command with ExecuteSQL but it doesn't seem to work
and the query pulls all of the records. Is it supported or will another
processor type allow me to LIMIT the result set returned?

Thanks,
Mike

Custom processors in MiNiFi

[Aakash Khochare]

Greetings,

While I understand that the primary use of NiFi/MiNiFi is for secure data 
ingress with the added benefit of Provenance, what are the views of the 
community on writing Processors that implement Machine Learning Algorithms and 
distribute them across Edge+ Cloud using NiFi and MiNiFi? Has anyone tried 
writing such processors?


Regards,

Aakash Khochare

Re: How can i compare hours in which data having with current datetime hours?

[prabhu Mahendran]

Mark,

Thanks for your information.

It works for my one input.

But i have another one input which have *'07/01/2017 0-1' *column.


07/01/2017 0-1,Nick,22

.

.

07/01/2012 23-24,Sandy,35



Consider my input is as above, how to match last 48 hours record from the
current hour in nifi?

On Mon, Jan 23, 2017 at 8:23 PM, Mark Payne  wrote:

> Prabhu,
>
> I think the RouteText processor will give you what you need. If you set
> the "Matching Strategy" property to "Satisfies Expression,"
> then it will allow you to use the Expression Language to evaluate each
> line of text in the file. Each line of text is available using
> the "line" variable. So, for example, you could create a Property named
> "older.than.72.hours" with a value of something like:
> ${line:getDelineatedValues(2):toDate('MM.dd.
> HH:mm:ss'):toNumber():lt( ${now():minus(25920)} )}
>
> The now():minus(25920) means subtract 25920 milliseconds (72
> hours) from the current date/time.
>
> So this will look at each line in the text file, get the second delineated
> value (by default it uses a comma as the delimiter), convert
> it to a date/time using the format of ..<4 digit
> year> <24-hour>::, and then convert
> that to a number (in milliseconds since midnight Jan. 1, 1970). It then
> compares this value to see if the value is less than the current
> date/time minus 72 hours.
>
> Each line of text that matches will be routed to 'older.than.72.hours'.
> Note, the outgoing FlowFiles may contain several lines of text
> each, though. It does not split the output into individual lines. You can
> do so if necessary via SplitText.
>
> Thanks!
> -Mark
>
>
> On Jan 23, 2017, at 5:23 AM, prabhu Mahendran 
> wrote:
>
> My input contains dateTime with hours like below.,
>
> *No,DateTime,Name,ID*
> *1,23.01.2016 09:02:21,Mega,201*
> *2,03.01.2016 10:02:23,Hema,202*
>
> Now i need to get 2nd Column ["02.01.2016 10:02:23"] and then compare it
> with current dateTime with milliseconds.
>
> After that if 2nd Column is in between past 72 hours and then insert into
> sql server.
>
> *For example.,*
>
>  My Current DateTime is "*04.01.2016 10:23:21"*
>
> For 1st row.,
>
>  1,*23.01.2016 09:02:21*,Mega,201
>
> =>Compare *23.01.2016 09:02:21* with 
> *04.01.2016
> 10:23:21 [Current Time].It couldn't insert into SQL Server due to *
> *DateTime*
> * is not past 72 hrs when it
>compared  with current dateTime.   *
> For 2 nd row,
>
>
>
>
>
> *2,03.01.2016 10:02:23,Hema,202=>This
> Should be insert into SQL Server.Because it's date is "03.01.2016" and it
> is past 72 hour data by compare it with current datetime. *
>
> *How can i achieve my use case in Nifi Processors?*
>
> *Please stop me if anything am doing wrong,*
>
> *Thanks,*
> *prabhu*
>
>
>

Re: Convert xls to csv in Nifi.

[prabhu Mahendran]

Can I expect the HSSF implementation to the existing PR?

On Fri, Jan 27, 2017 at 10:06 AM, prabhu Mahendran 
wrote:

> Jeremy,Thanks for your information.
>
> i think you will make effort to HSSF implementation in PR for convert (XLS
> into CSV). Is it right?
>
> Many thanks,
> prabhu
>
> On Wed, Jan 25, 2017 at 7:01 PM, Jeremy Dyer  wrote:
>
>> Prabhu - NIFI-2613 is currently only able to convert .xlxs documents to
>> csv. The processor uses Apache POI XSSF implementation which only supports
>> xlxs while HSSF would be pre 2007 excel files. I think to your point I
>> should probably make an effort to add the HSSF implementation to the
>> existing PR.
>>
>> - Jeremy
>>
>> On Wed, Jan 25, 2017 at 1:31 AM, prabhu Mahendran <
>> prabhuu161...@gmail.com> wrote:
>>
>>> Hi All,
>>>
>>> i have look into below JIRA  for conversion of my excel documents into
>>> csv file.
>>>
>>> https://issues.apache.org/jira/browse/NIFI-2613
>>>
>>> i have apply patches in GitHub Pull request.And then i can able to
>>> convert my .xlxs documents into csv file.
>>>
>>> But while give .xls documents it can't converted to csv file.
>>>
>>> Is patch applicable in jira  only for .xlxs files or it's too for all
>>> excel formats?
>>>
>>> Many thanks,
>>> prabhu
>>>
>>
>>
>