Re: Processor missing when attempting Add Processor

2017-03-28 Thread Joseph Niemiec
Just a simple idea, but try clearing your browser cache out and reloading
the entire canvas. I have had some funny behaviours before with a stale
cache not seeing anything.

On Tue, Mar 28, 2017 at 5:57 PM, Russell Bateman 
wrote:

> I've built a NAR containing a custom processor that loads in NiFi, but the
> processor cannot be found. At the top of /TikaProcessor.java/, I have,
> among other annotations, this:
>
> @Tags( { "tika" } )
>
> I've tried a number of things to solve this, like making it the only NAR
> besides just the set NiFi 1.1.1 ships with, and wiping out all flows down
> to a blank canvas. I tried removing SNAPSHOT from its version. *I see this
> in **/logs/nifi-app.log/*: This is the only reference to it in any log, but
> it does make me think that it's loaded:
>
> 2017-03-28 15:24:20,226 INFO [main] org.apache.nifi.nar.NarClassLoaders
> Loaded NAR file: /home/russ/dev/nifi/nifi-1.1.1
> /./work/nar/extensions/nifi-tika-1.0.1.nar-unpacked as class loader
> org.apache.nifi.nar.NarClassLoader[./work/nar/extensions/
> nifi-tika-1.0.1.nar-unpacked]
>
> In /resources/META_INF/services/, /org.apache.nifi.processor.Processor/
> contains:
>
> /com.imatsolutions.nifi.processor.TikaProcessor/
>
>  I have a separate, very large project with many custom processors all of
> which load fine (and I've used this one before. In fact, the only thing
> I've done is remove this long-working processor from a larger set.) This
> project is very small:
>
>nifi-tika
>+-- nar
>| +-- /nifi-tika-1.0.1.nar/
>|   `-- pom.xml (packaging is "nar")
>+-- pom.xml (packaging is "pom")
>`-- tika
> +-- pom.xml (packaging is "jar")
> +-- src (test, resources, etc. including
>/resources/META_INF/services/org.apache.nifi.p//rocessor.Processor/)
> `-- target
> `-- /tika-1.0.1.jar/
>
> I'm not certain what else to try. My /pom.xml/ files produce a NAR. NiFi
> appears to dignify it as a NAR. I keep retracing all the steps, but cannot
> figure out what I've missed.
>
>


-- 
Joseph


Re: Processor missing when attempting Add Processor

2017-03-28 Thread Joseph Niemiec
Sure please share, I have alittle time tonight.

On Tue, Mar 28, 2017 at 6:53 PM, Russell Bateman 
wrote:

> Thanks, I have tried this on both Firefox and Chrome. (I'm on Linux.) This
> is a very perplexing problem because it's so simple. I'm happy to make
> available /pom.xml/ files and/or the NAR.
>
> On 03/28/2017 04:31 PM, Joseph Niemiec wrote:
>
>> Just a simple idea, but try clearing your browser cache out and reloading
>> the entire canvas. I have had some funny behaviours before with a stale
>> cache not seeing anything.
>>
>> On Tue, Mar 28, 2017 at 5:57 PM, Russell Bateman 
>> wrote:
>>
>> I've built a NAR containing a custom processor that loads in NiFi, but the
>>> processor cannot be found. At the top of /TikaProcessor.java/, I have,
>>> among other annotations, this:
>>>
>>> @Tags( { "tika" } )
>>>
>>> I've tried a number of things to solve this, like making it the only NAR
>>> besides just the set NiFi 1.1.1 ships with, and wiping out all flows down
>>> to a blank canvas. I tried removing SNAPSHOT from its version. *I see
>>> this
>>> in **/logs/nifi-app.log/*: This is the only reference to it in any log,
>>> but
>>> it does make me think that it's loaded:
>>>
>>> 2017-03-28 15:24:20,226 INFO [main] org.apache.nifi.nar.NarClassLoaders
>>> Loaded NAR file: /home/russ/dev/nifi/nifi-1.1.1
>>> /./work/nar/extensions/nifi-tika-1.0.1.nar-unpacked as class loader
>>> org.apache.nifi.nar.NarClassLoader[./work/nar/extensions/
>>> nifi-tika-1.0.1.nar-unpacked]
>>>
>>> In /resources/META_INF/services/, /org.apache.nifi.processor.Processor/
>>> contains:
>>>
>>> /com.imatsolutions.nifi.processor.TikaProcessor/
>>>
>>>   I have a separate, very large project with many custom processors all
>>> of
>>> which load fine (and I've used this one before. In fact, the only thing
>>> I've done is remove this long-working processor from a larger set.) This
>>> project is very small:
>>>
>>> nifi-tika
>>> +-- nar
>>> | +-- /nifi-tika-1.0.1.nar/
>>> |   `-- pom.xml (packaging is "nar")
>>> +-- pom.xml (packaging is "pom")
>>> `-- tika
>>>  +-- pom.xml (packaging is "jar")
>>>  +-- src (test, resources, etc. including
>>> /resources/META_INF/services/org.apache.nifi.p//rocessor.Processor/)
>>>  `-- target
>>>  `-- /tika-1.0.1.jar/
>>>
>>> I'm not certain what else to try. My /pom.xml/ files produce a NAR. NiFi
>>> appears to dignify it as a NAR. I keep retracing all the steps, but
>>> cannot
>>> figure out what I've missed.
>>>
>>>
>>>
>>
>


-- 
Joseph


Re: Processor missing when attempting Add Processor

2017-03-28 Thread Joseph Niemiec
So just exploring the nars I was unable to find any class or resources or
services folder just maven bundled dependency and the package folder name
but not the .class file.

Can you share the content of
your  /src/main/resources/META-INF/services/org.apache.nifi.processor.Processor
file?

You also have this entry in some pom's which I dont have in my NAR projects.



org.apache.nifi
nifi-nar-maven-plugin
1.0.1-incubating
true



On Tue, Mar 28, 2017 at 7:09 PM, Russell Bateman 
wrote:

> Thanks! I'll write some kind of blog post on this once we figure it out.
>
> http://www.windofkeltia.com/preview/
>
>
> On 03/28/2017 04:59 PM, Joseph Niemiec wrote:
>
>> Sure please share, I have alittle time tonight.
>>
>> On Tue, Mar 28, 2017 at 6:53 PM, Russell Bateman 
>> wrote:
>>
>> Thanks, I have tried this on both Firefox and Chrome. (I'm on Linux.) This
>>> is a very perplexing problem because it's so simple. I'm happy to make
>>> available /pom.xml/ files and/or the NAR.
>>>
>>> On 03/28/2017 04:31 PM, Joseph Niemiec wrote:
>>>
>>> Just a simple idea, but try clearing your browser cache out and reloading
>>>> the entire canvas. I have had some funny behaviours before with a stale
>>>> cache not seeing anything.
>>>>
>>>> On Tue, Mar 28, 2017 at 5:57 PM, Russell Bateman >>> >
>>>> wrote:
>>>>
>>>> I've built a NAR containing a custom processor that loads in NiFi, but
>>>> the
>>>>
>>>>> processor cannot be found. At the top of /TikaProcessor.java/, I have,
>>>>> among other annotations, this:
>>>>>
>>>>> @Tags( { "tika" } )
>>>>>
>>>>> I've tried a number of things to solve this, like making it the only
>>>>> NAR
>>>>> besides just the set NiFi 1.1.1 ships with, and wiping out all flows
>>>>> down
>>>>> to a blank canvas. I tried removing SNAPSHOT from its version. *I see
>>>>> this
>>>>> in **/logs/nifi-app.log/*: This is the only reference to it in any log,
>>>>> but
>>>>> it does make me think that it's loaded:
>>>>>
>>>>> 2017-03-28 15:24:20,226 INFO [main] org.apache.nifi.nar.NarClassLo
>>>>> aders
>>>>> Loaded NAR file: /home/russ/dev/nifi/nifi-1.1.1
>>>>> /./work/nar/extensions/nifi-tika-1.0.1.nar-unpacked as class loader
>>>>> org.apache.nifi.nar.NarClassLoader[./work/nar/extensions/
>>>>> nifi-tika-1.0.1.nar-unpacked]
>>>>>
>>>>> In /resources/META_INF/services/, /org.apache.nifi.processor.Pro
>>>>> cessor/
>>>>> contains:
>>>>>
>>>>> /com.imatsolutions.nifi.processor.TikaProcessor/
>>>>>
>>>>>I have a separate, very large project with many custom processors
>>>>> all
>>>>> of
>>>>> which load fine (and I've used this one before. In fact, the only thing
>>>>> I've done is remove this long-working processor from a larger set.)
>>>>> This
>>>>> project is very small:
>>>>>
>>>>>  nifi-tika
>>>>>  +-- nar
>>>>>  | +-- /nifi-tika-1.0.1.nar/
>>>>>  |   `-- pom.xml (packaging is "nar")
>>>>>  +-- pom.xml (packaging is "pom")
>>>>>  `-- tika
>>>>>   +-- pom.xml (packaging is "jar")
>>>>>   +-- src (test, resources, etc. including
>>>>>  /resources/META_INF/services/org.apache.nifi.p//rocessor.Pr
>>>>> ocessor/)
>>>>>   `-- target
>>>>>   `-- /tika-1.0.1.jar/
>>>>>
>>>>> I'm not certain what else to try. My /pom.xml/ files produce a NAR.
>>>>> NiFi
>>>>> appears to dignify it as a NAR. I keep retracing all the steps, but
>>>>> cannot
>>>>> figure out what I've missed.
>>>>>
>>>>>
>>>>>
>>>>>
>>
>


-- 
Joseph


Re: Processor missing when attempting Add Processor

2017-03-28 Thread Joseph Niemiec
Excuse my last email I am blind to both apparently. I found the resources.
Going to test deploy it now.

On Tue, Mar 28, 2017 at 7:38 PM, Russell Bateman 
wrote:

> Thanks.
>
> /src/main/resources/META-INF/services/org.apache.nifi.processor.Processor/
> should contain (as originally noted):
>
> com.imatsolutions.nifi.processor.TikaProcessor
>
> I use the /nifi-nar-maven-plugin/ to manufacture the NAR from the JAR. I
> will double-check to see if the META-INF stuff didn't make it into the JAR
> (and thence into the NAR).
>
>
>
> On 03/28/2017 05:34 PM, Joseph Niemiec wrote:
>
>> So just exploring the nars I was unable to find any class or resources or
>> services folder just maven bundled dependency and the package folder name
>> but not the .class file.
>>
>> Can you share the content of
>> your  /src/main/resources/META-INF/services/org.apache.nifi.proces
>> sor.Processor
>> file?
>>
>> You also have this entry in some pom's which I dont have in my NAR
>> projects.
>>
>> 
>> 
>> org.apache.nifi
>> nifi-nar-maven-plugin
>> 1.0.1-incubating
>> true
>> 
>>
>>
>> On Tue, Mar 28, 2017 at 7:09 PM, Russell Bateman 
>> wrote:
>>
>> Thanks! I'll write some kind of blog post on this once we figure it out.
>>>
>>> http://www.windofkeltia.com/preview/
>>>
>>>
>>> On 03/28/2017 04:59 PM, Joseph Niemiec wrote:
>>>
>>> Sure please share, I have alittle time tonight.
>>>>
>>>> On Tue, Mar 28, 2017 at 6:53 PM, Russell Bateman >>> >
>>>> wrote:
>>>>
>>>> Thanks, I have tried this on both Firefox and Chrome. (I'm on Linux.)
>>>> This
>>>>
>>>>> is a very perplexing problem because it's so simple. I'm happy to make
>>>>> available /pom.xml/ files and/or the NAR.
>>>>>
>>>>> On 03/28/2017 04:31 PM, Joseph Niemiec wrote:
>>>>>
>>>>> Just a simple idea, but try clearing your browser cache out and
>>>>> reloading
>>>>>
>>>>>> the entire canvas. I have had some funny behaviours before with a
>>>>>> stale
>>>>>> cache not seeing anything.
>>>>>>
>>>>>> On Tue, Mar 28, 2017 at 5:57 PM, Russell Bateman <
>>>>>> r...@windofkeltia.com
>>>>>> wrote:
>>>>>>
>>>>>> I've built a NAR containing a custom processor that loads in NiFi, but
>>>>>> the
>>>>>>
>>>>>> processor cannot be found. At the top of /TikaProcessor.java/, I have,
>>>>>>> among other annotations, this:
>>>>>>>
>>>>>>> @Tags( { "tika" } )
>>>>>>>
>>>>>>> I've tried a number of things to solve this, like making it the only
>>>>>>> NAR
>>>>>>> besides just the set NiFi 1.1.1 ships with, and wiping out all flows
>>>>>>> down
>>>>>>> to a blank canvas. I tried removing SNAPSHOT from its version. *I see
>>>>>>> this
>>>>>>> in **/logs/nifi-app.log/*: This is the only reference to it in any
>>>>>>> log,
>>>>>>> but
>>>>>>> it does make me think that it's loaded:
>>>>>>>
>>>>>>> 2017-03-28 15:24:20,226 INFO [main] org.apache.nifi.nar.NarClassLo
>>>>>>> aders
>>>>>>> Loaded NAR file: /home/russ/dev/nifi/nifi-1.1.1
>>>>>>> /./work/nar/extensions/nifi-tika-1.0.1.nar-unpacked as class loader
>>>>>>> org.apache.nifi.nar.NarClassLoader[./work/nar/extensions/
>>>>>>> nifi-tika-1.0.1.nar-unpacked]
>>>>>>>
>>>>>>> In /resources/META_INF/services/, /org.apache.nifi.processor.Pro
>>>>>>> cessor/
>>>>>>> contains:
>>>>>>>
>>>>>>> /com.imatsolutions.nifi.processor.TikaProcessor/
>>>>>>>
>>>>>>> I have a separate, very large project with many custom processors
>>>>>>> all
>>>>>>> of
>>>>>>> which load fine (and I've used this one before. In fact, the only
>>>>>>> thing
>>>>>>> I've done is remove this long-working processor from a larger set.)
>>>>>>> This
>>>>>>> project is very small:
>>>>>>>
>>>>>>>   nifi-tika
>>>>>>>   +-- nar
>>>>>>>   | +-- /nifi-tika-1.0.1.nar/
>>>>>>>   |   `-- pom.xml (packaging is "nar")
>>>>>>>   +-- pom.xml (packaging is "pom")
>>>>>>>   `-- tika
>>>>>>>+-- pom.xml (packaging is "jar")
>>>>>>>+-- src (test, resources, etc. including
>>>>>>>   /resources/META_INF/services/org.apache.nifi.p//rocessor.Pr
>>>>>>> ocessor/)
>>>>>>>`-- target
>>>>>>>`-- /tika-1.0.1.jar/
>>>>>>>
>>>>>>> I'm not certain what else to try. My /pom.xml/ files produce a NAR.
>>>>>>> NiFi
>>>>>>> appears to dignify it as a NAR. I keep retracing all the steps, but
>>>>>>> cannot
>>>>>>> figure out what I've missed.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>
>


-- 
Joseph


Re: Processor missing when attempting Add Processor

2017-03-28 Thread Joseph Niemiec
its called tika-1.0.1.jar without the nifi prefix in the bundled depnds.

On Tue, Mar 28, 2017 at 7:48 PM, Russell Bateman 
wrote:

> Joseph,
>
> You're right. Somehow, /nifi-tika-1.0.1.jar/ is not found among the myriad
> JARs under /META-INF/bundle//d-dependen//c//ies/ in the NAR. I thought
> I'd seen it in there, but I had not looked hard enough. I'll have to
> revisit another (working) /pom.xml/ to see how that's stuffed into it
> (mysterious since all the other dependencies got in).
>
> Yes, I had once found a later version of the /nifi-nar-plugin/ and tried
> it (because I tend to go with the latest tools for things like this), but
> without success so I went back to this one, the /1.0.1-incubating/ plug-in,
> which is the one we started using at the beginning (about 18 months ago)
> and have always used.
>
> Thanks very much for the second pair of eyes. I will insist upon checking
> for this in my up-coming post, etc.
>
> Russ
>
> On 03/28/2017 05:34 PM, Joseph Niemiec wrote:
>
>> So just exploring the nars I was unable to find any class or resources or
>> services folder just maven bundled dependency and the package folder name
>> but not the .class file.
>>
>> Can you share the content of
>> your  /src/main/resources/META-INF/services/org.apache.nifi.proces
>> sor.Processor
>> file?
>>
>> You also have this entry in some pom's which I dont have in my NAR
>> projects.
>>
>> 
>> 
>> org.apache.nifi
>> nifi-nar-maven-plugin
>> 1.0.1-incubating
>> true
>> 
>>
>>
>> On Tue, Mar 28, 2017 at 7:09 PM, Russell Bateman 
>> wrote:
>>
>> Thanks! I'll write some kind of blog post on this once we figure it out.
>>>
>>> http://www.windofkeltia.com/preview/
>>>
>>>
>>> On 03/28/2017 04:59 PM, Joseph Niemiec wrote:
>>>
>>> Sure please share, I have alittle time tonight.
>>>>
>>>> On Tue, Mar 28, 2017 at 6:53 PM, Russell Bateman >>> >
>>>> wrote:
>>>>
>>>> Thanks, I have tried this on both Firefox and Chrome. (I'm on Linux.)
>>>> This
>>>>
>>>>> is a very perplexing problem because it's so simple. I'm happy to make
>>>>> available /pom.xml/ files and/or the NAR.
>>>>>
>>>>> On 03/28/2017 04:31 PM, Joseph Niemiec wrote:
>>>>>
>>>>> Just a simple idea, but try clearing your browser cache out and
>>>>> reloading
>>>>>
>>>>>> the entire canvas. I have had some funny behaviours before with a
>>>>>> stale
>>>>>> cache not seeing anything.
>>>>>>
>>>>>> On Tue, Mar 28, 2017 at 5:57 PM, Russell Bateman <
>>>>>> r...@windofkeltia.com
>>>>>> wrote:
>>>>>>
>>>>>> I've built a NAR containing a custom processor that loads in NiFi, but
>>>>>> the
>>>>>>
>>>>>> processor cannot be found. At the top of /TikaProcessor.java/, I have,
>>>>>>> among other annotations, this:
>>>>>>>
>>>>>>> @Tags( { "tika" } )
>>>>>>>
>>>>>>> I've tried a number of things to solve this, like making it the only
>>>>>>> NAR
>>>>>>> besides just the set NiFi 1.1.1 ships with, and wiping out all flows
>>>>>>> down
>>>>>>> to a blank canvas. I tried removing SNAPSHOT from its version. *I see
>>>>>>> this
>>>>>>> in **/logs/nifi-app.log/*: This is the only reference to it in any
>>>>>>> log,
>>>>>>> but
>>>>>>> it does make me think that it's loaded:
>>>>>>>
>>>>>>> 2017-03-28 15:24:20,226 INFO [main] org.apache.nifi.nar.NarClassLo
>>>>>>> aders
>>>>>>> Loaded NAR file: /home/russ/dev/nifi/nifi-1.1.1
>>>>>>> /./work/nar/extensions/nifi-tika-1.0.1.nar-unpacked as class loader
>>>>>>> org.apache.nifi.nar.NarClassLoader[./work/nar/extensions/
>>>>>>> nifi-tika-1.0.1.nar-unpacked]
>>>>>>>
>>>>>>> In /resources/META_INF/services/, /org.apache.nifi.processor.Pro
>>>>>>> cessor/
>>>>>>> contains:
>>>>>>>
>>>>>>> /com.imatsolutions.nifi.processor.TikaProcessor/
>>>>>>>
>>>>>>> I have a separate, very large project with many custom processors
>>>>>>> all
>>>>>>> of
>>>>>>> which load fine (and I've used this one before. In fact, the only
>>>>>>> thing
>>>>>>> I've done is remove this long-working processor from a larger set.)
>>>>>>> This
>>>>>>> project is very small:
>>>>>>>
>>>>>>>   nifi-tika
>>>>>>>   +-- nar
>>>>>>>   | +-- /nifi-tika-1.0.1.nar/
>>>>>>>   |   `-- pom.xml (packaging is "nar")
>>>>>>>   +-- pom.xml (packaging is "pom")
>>>>>>>   `-- tika
>>>>>>>+-- pom.xml (packaging is "jar")
>>>>>>>+-- src (test, resources, etc. including
>>>>>>>   /resources/META_INF/services/org.apache.nifi.p//rocessor.Pr
>>>>>>> ocessor/)
>>>>>>>`-- target
>>>>>>>`-- /tika-1.0.1.jar/
>>>>>>>
>>>>>>> I'm not certain what else to try. My /pom.xml/ files produce a NAR.
>>>>>>> NiFi
>>>>>>> appears to dignify it as a NAR. I keep retracing all the steps, but
>>>>>>> cannot
>>>>>>> figure out what I've missed.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>
>


-- 
Joseph


Re: Processor missing when attempting Add Processor

2017-03-28 Thread Joseph Niemiec
So once deployed my UI also does not display anything. So I dug deeper into
your tika-1.0.1.jar file and found inside the processors folder your
TikaProcess.class along with 2 innerclasses called $1 and $2. I don't have
this one though in my nars. So I tried to look around for your Pom plugin.

Would you try rebuilding and updating your pom plugin to the current dev
doc recommendation (drop the incubator).

https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#nars




org.apache.nifi
nifi-nar-maven-plugin
1.1.0
true





On Tue, Mar 28, 2017 at 7:50 PM, Joseph Niemiec 
wrote:

> its called tika-1.0.1.jar without the nifi prefix in the bundled depnds.
>
> On Tue, Mar 28, 2017 at 7:48 PM, Russell Bateman 
> wrote:
>
>> Joseph,
>>
>> You're right. Somehow, /nifi-tika-1.0.1.jar/ is not found among the
>> myriad JARs under /META-INF/bundle//d-dependen//c//ies/ in the NAR. I
>> thought I'd seen it in there, but I had not looked hard enough. I'll have
>> to revisit another (working) /pom.xml/ to see how that's stuffed into it
>> (mysterious since all the other dependencies got in).
>>
>> Yes, I had once found a later version of the /nifi-nar-plugin/ and tried
>> it (because I tend to go with the latest tools for things like this), but
>> without success so I went back to this one, the /1.0.1-incubating/ plug-in,
>> which is the one we started using at the beginning (about 18 months ago)
>> and have always used.
>>
>> Thanks very much for the second pair of eyes. I will insist upon checking
>> for this in my up-coming post, etc.
>>
>> Russ
>>
>> On 03/28/2017 05:34 PM, Joseph Niemiec wrote:
>>
>>> So just exploring the nars I was unable to find any class or resources or
>>> services folder just maven bundled dependency and the package folder name
>>> but not the .class file.
>>>
>>> Can you share the content of
>>> your  /src/main/resources/META-INF/services/org.apache.nifi.proces
>>> sor.Processor
>>> file?
>>>
>>> You also have this entry in some pom's which I dont have in my NAR
>>> projects.
>>>
>>> 
>>> 
>>> org.apache.nifi
>>> nifi-nar-maven-plugin
>>> 1.0.1-incubating
>>> true
>>> 
>>>
>>>
>>> On Tue, Mar 28, 2017 at 7:09 PM, Russell Bateman 
>>> wrote:
>>>
>>> Thanks! I'll write some kind of blog post on this once we figure it out.
>>>>
>>>> http://www.windofkeltia.com/preview/
>>>>
>>>>
>>>> On 03/28/2017 04:59 PM, Joseph Niemiec wrote:
>>>>
>>>> Sure please share, I have alittle time tonight.
>>>>>
>>>>> On Tue, Mar 28, 2017 at 6:53 PM, Russell Bateman <
>>>>> r...@windofkeltia.com>
>>>>> wrote:
>>>>>
>>>>> Thanks, I have tried this on both Firefox and Chrome. (I'm on Linux.)
>>>>> This
>>>>>
>>>>>> is a very perplexing problem because it's so simple. I'm happy to make
>>>>>> available /pom.xml/ files and/or the NAR.
>>>>>>
>>>>>> On 03/28/2017 04:31 PM, Joseph Niemiec wrote:
>>>>>>
>>>>>> Just a simple idea, but try clearing your browser cache out and
>>>>>> reloading
>>>>>>
>>>>>>> the entire canvas. I have had some funny behaviours before with a
>>>>>>> stale
>>>>>>> cache not seeing anything.
>>>>>>>
>>>>>>> On Tue, Mar 28, 2017 at 5:57 PM, Russell Bateman <
>>>>>>> r...@windofkeltia.com
>>>>>>> wrote:
>>>>>>>
>>>>>>> I've built a NAR containing a custom processor that loads in NiFi,
>>>>>>> but
>>>>>>> the
>>>>>>>
>>>>>>> processor cannot be found. At the top of /TikaProcessor.java/, I
>>>>>>>> have,
>>>>>>>> among other annotations, this:
>>>>>>>>
>>>>>>>> @Tags( { "tika" } )
>>>>>>>>
>>>>>>>> I've tried a number of things to solve this, like making it the only
>>>>>>>> NAR
>>>>>>>> besides just the set NiFi 1.1.1 ships with, and wiping out all flows
>>>>>

Re: Processor missing when attempting Add Processor

2017-03-28 Thread Joseph Niemiec
Can you please also update the Nifi Version in your parent pom as well to
1.1.1 like your attempting to run on, its currently set for 1.0.0.

*1.0.0*
1.7.214.12truetrue..

1.0.01.7.214.12truetrue..

On Tue, Mar 28, 2017 at 8:09 PM, Joseph Niemiec 
wrote:

> So once deployed my UI also does not display anything. So I dug deeper
> into your tika-1.0.1.jar file and found inside the processors folder your
> TikaProcess.class along with 2 innerclasses called $1 and $2. I don't have
> this one though in my nars. So I tried to look around for your Pom plugin.
>
> Would you try rebuilding and updating your pom plugin to the current dev
> doc recommendation (drop the incubator).
>
> https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#nars
>
> 
> 
> 
> org.apache.nifi
> nifi-nar-maven-plugin
> 1.1.0
> true
>     
> 
> 
>
>
> On Tue, Mar 28, 2017 at 7:50 PM, Joseph Niemiec 
> wrote:
>
>> its called tika-1.0.1.jar without the nifi prefix in the bundled depnds.
>>
>> On Tue, Mar 28, 2017 at 7:48 PM, Russell Bateman 
>> wrote:
>>
>>> Joseph,
>>>
>>> You're right. Somehow, /nifi-tika-1.0.1.jar/ is not found among the
>>> myriad JARs under /META-INF/bundle//d-dependen//c//ies/ in the NAR. I
>>> thought I'd seen it in there, but I had not looked hard enough. I'll have
>>> to revisit another (working) /pom.xml/ to see how that's stuffed into it
>>> (mysterious since all the other dependencies got in).
>>>
>>> Yes, I had once found a later version of the /nifi-nar-plugin/ and tried
>>> it (because I tend to go with the latest tools for things like this), but
>>> without success so I went back to this one, the /1.0.1-incubating/ plug-in,
>>> which is the one we started using at the beginning (about 18 months ago)
>>> and have always used.
>>>
>>> Thanks very much for the second pair of eyes. I will insist upon
>>> checking for this in my up-coming post, etc.
>>>
>>> Russ
>>>
>>> On 03/28/2017 05:34 PM, Joseph Niemiec wrote:
>>>
>>>> So just exploring the nars I was unable to find any class or resources
>>>> or
>>>> services folder just maven bundled dependency and the package folder
>>>> name
>>>> but not the .class file.
>>>>
>>>> Can you share the content of
>>>> your  /src/main/resources/META-INF/services/org.apache.nifi.proces
>>>> sor.Processor
>>>> file?
>>>>
>>>> You also have this entry in some pom's which I dont have in my NAR
>>>> projects.
>>>>
>>>> 
>>>> 
>>>> org.apache.nifi
>>>> nifi-nar-maven-plugin
>>>> 1.0.1-incubating
>>>> true
>>>> 
>>>>
>>>>
>>>> On Tue, Mar 28, 2017 at 7:09 PM, Russell Bateman >>> >
>>>> wrote:
>>>>
>>>> Thanks! I'll write some kind of blog post on this once we figure it out.
>>>>>
>>>>> http://www.windofkeltia.com/preview/
>>>>>
>>>>>
>>>>> On 03/28/2017 04:59 PM, Joseph Niemiec wrote:
>>>>>
>>>>> Sure please share, I have alittle time tonight.
>>>>>>
>>>>>> On Tue, Mar 28, 2017 at 6:53 PM, Russell Bateman <
>>>>>> r...@windofkeltia.com>
>>>>>> wrote:
>>>>>>
>>>>>> Thanks, I have tried this on both Firefox and Chrome. (I'm on Linux.)
>>>>>> This
>>>>>>
>>>>>>> is a very perplexing problem because it's so simple. I'm happy to
>>>>>>> make
>>>>>>> available /pom.xml/ files and/or the NAR.
>>>>>>>
>>>>>>> On 03/28/2017 04:31 PM, Joseph Niemiec wrote:
>>>>>>>
>>>>>>> Just a simple idea, but try clearing your browser cache out and
>>>>>>> reloading
>>>>>>>
>>>>>>>> the entire canvas. I have had some funny behaviours before with a
>>>>>>>> stale
>>>>>>>> cache not seeing anything.
>>>>>>>>
>>>>>>>> On Tue, Mar 28, 2017 at 5:57 PM, Russell Bateman <
>>>>>>>> r...@windofkeltia.com
>>>>>>>> wrote:
>>>>>>>>
>>>>

Re: Processor missing when attempting Add Processor

2017-03-28 Thread Joseph Niemiec
The packaged utils is from 1.0.1 and should read 1.1.1 inside the nars
packaged maven depends. Looking at the new parent pom can you change the
last zero to a 1 ?

1.0.1

to

1.1.1


On Tue, Mar 28, 2017 at 8:27 PM, Russell Bateman 
wrote:

> Both pom.xml and NAR updated (and still not working) on
> windofkeltia.com/preview.
>
> On 03/28/2017 06:22 PM, Joseph Niemiec wrote:
>
>> Can you please also update the Nifi Version in your parent pom as well to
>> 1.1.1 like your attempting to run on, its currently set for 1.0.0.
>>
>> *1.0.0*
>>
>> 1.7.21> ersion>4.12true> install.skip>true<
>> code.root>..
>>
>> 1.0.0> >1.7.214.12<
>> maven.install.skip>true> deploy.skip>true..> root>
>>
>> On Tue, Mar 28, 2017 at 8:09 PM, Joseph Niemiec 
>> wrote:
>>
>> So once deployed my UI also does not display anything. So I dug deeper
>>> into your tika-1.0.1.jar file and found inside the processors folder your
>>> TikaProcess.class along with 2 innerclasses called $1 and $2. I don't
>>> have
>>> this one though in my nars. So I tried to look around for your Pom
>>> plugin.
>>>
>>> Would you try rebuilding and updating your pom plugin to the current dev
>>> doc recommendation (drop the incubator).
>>>
>>> https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#nars
>>>
>>> 
>>>  
>>>  
>>>  org.apache.nifi
>>>  nifi-nar-maven-plugin
>>>  1.1.0
>>>  true
>>>  
>>>  
>>> 
>>>
>>>
>>> On Tue, Mar 28, 2017 at 7:50 PM, Joseph Niemiec 
>>> wrote:
>>>
>>> its called tika-1.0.1.jar without the nifi prefix in the bundled depnds.
>>>>
>>>> On Tue, Mar 28, 2017 at 7:48 PM, Russell Bateman >>> >
>>>> wrote:
>>>>
>>>> Joseph,
>>>>>
>>>>> You're right. Somehow, /nifi-tika-1.0.1.jar/ is not found among the
>>>>> myriad JARs under /META-INF/bundle//d-dependen//c//ies/ in the NAR. I
>>>>> thought I'd seen it in there, but I had not looked hard enough. I'll
>>>>> have
>>>>> to revisit another (working) /pom.xml/ to see how that's stuffed into
>>>>> it
>>>>> (mysterious since all the other dependencies got in).
>>>>>
>>>>> Yes, I had once found a later version of the /nifi-nar-plugin/ and
>>>>> tried
>>>>> it (because I tend to go with the latest tools for things like this),
>>>>> but
>>>>> without success so I went back to this one, the /1.0.1-incubating/
>>>>> plug-in,
>>>>> which is the one we started using at the beginning (about 18 months
>>>>> ago)
>>>>> and have always used.
>>>>>
>>>>> Thanks very much for the second pair of eyes. I will insist upon
>>>>> checking for this in my up-coming post, etc.
>>>>>
>>>>> Russ
>>>>>
>>>>> On 03/28/2017 05:34 PM, Joseph Niemiec wrote:
>>>>>
>>>>> So just exploring the nars I was unable to find any class or resources
>>>>>> or
>>>>>> services folder just maven bundled dependency and the package folder
>>>>>> name
>>>>>> but not the .class file.
>>>>>>
>>>>>> Can you share the content of
>>>>>> your  /src/main/resources/META-INF/services/org.apache.nifi.proces
>>>>>> sor.Processor
>>>>>> file?
>>>>>>
>>>>>> You also have this entry in some pom's which I dont have in my NAR
>>>>>> projects.
>>>>>>
>>>>>> 
>>>>>> 
>>>>>> org.apache.nifi
>>>>>> nifi-nar-maven-plugin
>>>>>> 1.0.1-incubating
>>>>>> true
>>>>>> 
>>>>>>
>>>>>>
>>>>>> On Tue, Mar 28, 2017 at 7:09 PM, Russell Bateman <
>>>>>> r...@windofkeltia.com
>>>>>> wrote:
>>>>>>
>>>>>> Thanks! I'll write some kind of blog post on this once we figure it
>>>>>> out.
>>>>>>
>>>>>>> http://www.windofkeltia.com/preview/
>>>>>>>
>&

Re: Processor missing when attempting Add Processor

2017-03-28 Thread Joseph Niemiec
ugghh.. Any objection to sharing your src project ziped/tared up?

On Tue, Mar 28, 2017 at 8:42 PM, Russell Bateman 
wrote:

> With these changes, I'm still not seeing /TikaProcessor/ in the list.
>
>
> On 03/28/2017 06:38 PM, Russell Bateman wrote:
>
>> (Sheesh. I apologize for trying to do something else concurrently. I
>> meant no disrespect. I'm on board now and the changes are up there again.
>> I'm awaiting NiFi to recycle now.)
>>
>> On 03/28/2017 06:33 PM, Joseph Niemiec wrote:
>>
>>> The packaged utils is from 1.0.1 and should read 1.1.1 inside the nars
>>> packaged maven depends. Looking at the new parent pom can you change the
>>> last zero to a 1 ?
>>>
>>> 1.0.1
>>>
>>> to
>>>
>>> 1.1.1
>>>
>>>
>>> On Tue, Mar 28, 2017 at 8:27 PM, Russell Bateman
>>> wrote:
>>>
>>> Both pom.xml and NAR updated (and still not working) on
>>>> windofkeltia.com/preview.
>>>>
>>>> On 03/28/2017 06:22 PM, Joseph Niemiec wrote:
>>>>
>>>> Can you please also update the Nifi Version in your parent pom as well
>>>>> to
>>>>> 1.1.1 like your attempting to run on, its currently set for 1.0.0.
>>>>>
>>>>> *1.0.0*
>>>>>
>>>>> 1.7.21>>>> ersion>4.12true>>>> install.skip>true<
>>>>> code.root>..
>>>>>
>>>>> 1.0.0>>>>
>>>>>> 1.7.214.12<
>>>>>>
>>>>> maven.install.skip>true>>>> deploy.skip>true..>>>> root>
>>>>>
>>>>> On Tue, Mar 28, 2017 at 8:09 PM, Joseph Niemiec
>>>>> wrote:
>>>>>
>>>>> So once deployed my UI also does not display anything. So I dug deeper
>>>>>
>>>>>> into your tika-1.0.1.jar file and found inside the processors folder
>>>>>> your
>>>>>> TikaProcess.class along with 2 innerclasses called $1 and $2. I don't
>>>>>> have
>>>>>> this one though in my nars. So I tried to look around for your Pom
>>>>>> plugin.
>>>>>>
>>>>>> Would you try rebuilding and updating your pom plugin to the current
>>>>>> dev
>>>>>> doc recommendation (drop the incubator).
>>>>>>
>>>>>> https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#nars
>>>>>>
>>>>>> 
>>>>>>   
>>>>>>   
>>>>>>   org.apache.nifi
>>>>>>   nifi-nar-maven-plugin
>>>>>>   1.1.0
>>>>>>   true
>>>>>>   
>>>>>>   
>>>>>> 
>>>>>>
>>>>>>
>>>>>> On Tue, Mar 28, 2017 at 7:50 PM, Joseph Niemiec
>>>>>> wrote:
>>>>>>
>>>>>> its called tika-1.0.1.jar without the nifi prefix in the bundled
>>>>>> depnds.
>>>>>>
>>>>>>> On Tue, Mar 28, 2017 at 7:48 PM, Russell Bateman <
>>>>>>> r...@windofkeltia.com
>>>>>>> wrote:
>>>>>>>
>>>>>>> Joseph,
>>>>>>>
>>>>>>>> You're right. Somehow, /nifi-tika-1.0.1.jar/ is not found among the
>>>>>>>> myriad JARs under /META-INF/bundle//d-dependen//c//ies/ in the
>>>>>>>> NAR. I
>>>>>>>> thought I'd seen it in there, but I had not looked hard enough. I'll
>>>>>>>> have
>>>>>>>> to revisit another (working) /pom.xml/ to see how that's stuffed
>>>>>>>> into
>>>>>>>> it
>>>>>>>> (mysterious since all the other dependencies got in).
>>>>>>>>
>>>>>>>> Yes, I had once found a later version of the /nifi-nar-plugin/ and
>>>>>>>> tried
>>>>>>>> it (because I tend to go with the latest tools for things like
>>>>>>>> this),
>>>>>>>> but
>>>>>>>> without success so I went back to this one, the /1.0.1-incubating/
>>>>>>>> plug-in,
>>>>>>>> whic

Re: NiFi Processors show 30 Second Execution time, 0 executions

2017-03-31 Thread Joseph Niemiec
Are you attempting to also attach a screenshot? My Email seems to think so
but no photo is coming out.  The zero tasks makes sense (see below) but I
am not 100% sure where your getting 30 seconds from, maybe another dev has
an idea on that one... Can we get a screenshot?

~~
Ran on 1.1.1...

Zero tasks makes sense as there has been no work scheduled for the
processor. Otherwise there is no need to run something like the
UpdateAttribute Processor nonstop each time. I went and turned the
processor logger to INFO from WARN (bottom of email) and you see the
Timer-Driven Process thread only log once a flowfile has landed into the
channel. Otherwise for 5 minutes after zipo if no new FlowFiles landed in
the channel.

If we look at the dev docs it mentions mostly the onTrigger and what 'work
scheduled means' (when processors are triggered)
https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#when-processors-are-triggered


Quote from the docs
"A Processor’s onTrigger method will be called only when it is scheduled to
run and when work exists for the Processor. Work is said to exist for a
Processor if any of the following conditions is met:
*A Connection whose destination is the Processor has at least one FlowFile
in its queue
*The Processors has no incoming Connections
*The Processor is annotated with the @TriggerWhenEmpty annotation"


If you keep going they talk about annotations, you could use the
onScheduled annotation if you always wanted to execute the task in a custom
processor.

#INFI UpdateAttribute Processor Logs
2017-03-31 08:21:22,393 INFO [StandardProcessScheduler Thread-3]
o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled
UpdateAttribute[id=24478d36-015b-1000-ce38-ca50c24234fd] to run with 1
threads
2017-03-31 08:21:22,570 INFO [Flow Service Tasks Thread-2]
o.a.nifi.controller.StandardFlowService Saved flow controller
org.apache.nifi.controller.FlowController@706b2b31 // Another save pending
= false
2017-03-31 08:21:34,666 INFO [StandardProcessScheduler Thread-6]
o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled
GenerateFlowFile[id=244774ca-015b-1000-5a1f-9cd57db06ec3] to run with 1
threads
2017-03-31 08:21:34,712 INFO [Timer-Driven Process Thread-5]
o.a.n.p.attributes.UpdateAttribute
UpdateAttribute[id=24478d36-015b-1000-ce38-ca50c24234fd] Updated attributes
for
StandardFlowFileRecord[uuid=74742b13-a650-4e19-a2e4-5a8adbfc259d,claim=,offset=0,name=82058058092238,size=0];
transferring to 'success'
2017-03-31 08:21:35,077 INFO [Flow Service Tasks Thread-2]
o.a.nifi.controller.StandardFlowService Saved flow controller
org.apache.nifi.controller.FlowController@706b2b31 // Another save pending
= false
2017-03-31 08:21:36,865 INFO [Provenance Maintenance Thread-2]
o.a.n.p.PersistentProvenanceRepository Created new Provenance Event Writers
for events starting with ID 8
2017-03-31 08:21:36,898 INFO [Provenance Repository Rollover Thread-1]
o.a.n.p.PersistentProvenanceRepository Successfully merged 16 journal files
(2 records) into single Provenance Log File ./provenance_repository/6.prov
in 34 milliseconds
2017-03-31 08:21:36,899 INFO [Provenance Repository Rollover Thread-1]
o.a.n.p.PersistentProvenanceRepository Successfully Rolled over Provenance
Event file containing 2 records. In the past 5 minutes, 2 events have been
written to the Provenance Repository, totaling 638 bytes
2017-03-31 08:21:37,960 INFO [NiFi Web Server-22]
o.a.n.controller.StandardProcessorNode Stopping processor: class
org.apache.nifi.processors.standard.GenerateFlowFile
2017-03-31 08:21:37,960 INFO [StandardProcessScheduler Thread-1]
o.a.n.c.s.TimerDrivenSchedulingAgent Stopped scheduling
GenerateFlowFile[id=244774ca-015b-1000-5a1f-9cd57db06ec3] to run
2017-03-31 08:21:38,081 INFO [Flow Service Tasks Thread-2]
o.a.nifi.controller.StandardFlowService Saved flow controller
org.apache.nifi.controller.FlowController@706b2b31 // Another save pending
= false
2017-03-31 08:21:46,956 INFO [Write-Ahead Local State Provider Maintenance]
org.wali.MinimalLockingWriteAheadLog
org.wali.MinimalLockingWriteAheadLog@7bebad0d checkpointed with 0 Records
and 0 Swap Files in 15 milliseconds (Stop-the-world time = 0 milliseconds,
Clear Edit Logs time = 0 millis), max Transaction ID -1
2017-03-31 08:21:47,253 INFO [pool-8-thread-1]
o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile
Repository
2017-03-31 08:21:47,303 INFO [pool-8-thread-1]
org.wali.MinimalLockingWriteAheadLog
org.wali.MinimalLockingWriteAheadLog@1dce481b checkpointed with 4 Records
and 0 Swap Files in 50 milliseconds (Stop-the-world time = 18 milliseconds,
Clear Edit Logs time = 12 millis), max Transaction ID 7
2017-03-31 08:21:47,304 INFO [pool-8-thread-1]
o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile
Repository with 4 records in 50 milliseconds


On Fri, Mar 31, 2017 at 4:25 AM, Peter Wicks (pwicks) 
wrote:

> I just noticed that on the latest version (which is now working for me)
> that all processors that have not recently run sho

Re: NiFi Processors show 30 Second Execution time, 0 executions

2017-03-31 Thread Joseph Niemiec
I just built the latest and am unable to see the issue as well. I also
played with yield duration with no luck.

Can you provide us what build your on so I can check that one out exactly?
I did my last trunk test as of - 6a64b3cd9cca70e6a27b9034eba520ae0c0cb6ca

git rev-parse HEAD


On Fri, Mar 31, 2017 at 9:21 AM, Lee Laim  wrote:

> Is it potentially related to yield duration?
>
>
>
> > On Mar 31, 2017, at 6:58 AM, Peter Wicks (pwicks) 
> wrote:
> >
> > channel.
>



-- 
Joseph


Re: NiFi Processors show 30 Second Execution time, 0 executions

2017-03-31 Thread Joseph Niemiec
Doing a clean build of 091359b450a7d0fb6bb04e2238c9171728cd2720, I will
have to see if I have a windows 7 VM anywhere, I know Witt was using Win10
and didnt see it... I find it odd that ALL your processors have a 30 second
number not just the UpdateAttribute. Anything else about your
environment you can share that may be unique?

On Fri, Mar 31, 2017 at 10:15 AM, Peter Wicks (pwicks) 
wrote:

> 091359b450a7d0fb6bb04e2238c9171728cd2720, so just one commit behind
> master.
> I am testing on Windows 7.
>
> Lee, yield isn't a bad idea, but UpdateAttribute in my screenshot has
> never run; not even once. I don't think it's had the opportunity to yield.
>
> -----Original Message-
> From: Joseph Niemiec [mailto:josephx...@gmail.com]
> Sent: Friday, March 31, 2017 9:37 PM
> To: dev@nifi.apache.org
> Subject: Re: NiFi Processors show 30 Second Execution time, 0 executions
>
> I just built the latest and am unable to see the issue as well. I also
> played with yield duration with no luck.
>
> Can you provide us what build your on so I can check that one out exactly?
> I did my last trunk test as of - 6a64b3cd9cca70e6a27b9034eba520ae0c0cb6ca
>
> git rev-parse HEAD
>
>
> On Fri, Mar 31, 2017 at 9:21 AM, Lee Laim  wrote:
>
> > Is it potentially related to yield duration?
> >
> >
> >
> > > On Mar 31, 2017, at 6:58 AM, Peter Wicks (pwicks)
> > > 
> > wrote:
> > >
> > > channel.
> >
>
>
>
> --
> Joseph
>



-- 
Joseph


Re: NiFi Processors show 30 Second Execution time, 0 executions

2017-03-31 Thread Joseph Niemiec
What version of Java are you running on ? Major_minor?

On Fri, Mar 31, 2017 at 10:49 AM, Peter Wicks (pwicks) 
wrote:

> I misread my own screenshot, it says 30 minutes, not seconds. Also, I did
> a restart of NiFi and opened it up in a fresh instance of Chrome; no
> change. I kicked off a GenerateFlowFile processor and the milliseconds are
> going up, but the 30 minutes is remaining the same...
>
>
>
> -Original Message-
> From: Joseph Niemiec [mailto:josephx...@gmail.com]
> Sent: Friday, March 31, 2017 10:41 PM
> To: dev@nifi.apache.org
> Subject: Re: NiFi Processors show 30 Second Execution time, 0 executions
>
> Doing a clean build of 091359b450a7d0fb6bb04e2238c9171728cd2720, I will
> have to see if I have a windows 7 VM anywhere, I know Witt was using Win10
> and didnt see it... I find it odd that ALL your processors have a 30 second
> number not just the UpdateAttribute. Anything else about your environment
> you can share that may be unique?
>
> On Fri, Mar 31, 2017 at 10:15 AM, Peter Wicks (pwicks) 
> wrote:
>
> > 091359b450a7d0fb6bb04e2238c9171728cd2720, so just one commit behind
> > master.
> > I am testing on Windows 7.
> >
> > Lee, yield isn't a bad idea, but UpdateAttribute in my screenshot has
> > never run; not even once. I don't think it's had the opportunity to
> yield.
> >
> > -Original Message-
> > From: Joseph Niemiec [mailto:josephx...@gmail.com]
> > Sent: Friday, March 31, 2017 9:37 PM
> > To: dev@nifi.apache.org
> > Subject: Re: NiFi Processors show 30 Second Execution time, 0
> > executions
> >
> > I just built the latest and am unable to see the issue as well. I also
> > played with yield duration with no luck.
> >
> > Can you provide us what build your on so I can check that one out
> exactly?
> > I did my last trunk test as of -
> > 6a64b3cd9cca70e6a27b9034eba520ae0c0cb6ca
> >
> > git rev-parse HEAD
> >
> >
> > On Fri, Mar 31, 2017 at 9:21 AM, Lee Laim  wrote:
> >
> > > Is it potentially related to yield duration?
> > >
> > >
> > >
> > > > On Mar 31, 2017, at 6:58 AM, Peter Wicks (pwicks)
> > > > 
> > > wrote:
> > > >
> > > > channel.
> > >
> >
> >
> >
> > --
> > Joseph
> >
>
>
>
> --
> Joseph
>



-- 
Joseph


Re: Deep Dive into NiFi's Site2Site

2017-04-04 Thread Joseph Niemiec
Not all of what your looking for but there are a number of details on the
HTTP implementation here. -
https://cwiki.apache.org/confluence/display/NIFI/Support+HTTP%28S%29+as+a+transport+mechanism+for+Site-to-Site


On Tue, Apr 4, 2017 at 5:32 AM, Pushkara R  wrote:

> Hi,
>
> I wanted to understand the site-to-site architecture of NiFi, mainly the
> differences between using HTTP and RAW protocols, the architecture of the
> site-to-site client and the input/output ports and how the NiFi REST API
> comes into picture in site-to-site.
>
> Could someone please point me to architecture documents or any other media
> that could clarify these things for me? If not, I will go through the code
> to understand these but I would really appreciate some pointers in that as
> well.
>
> Thanks a lot.
>



-- 
Joseph


Re: [DISCUSS] Component deprecation annotation

2017-05-01 Thread Joseph Niemiec
It would be really nice if this could be part of a summary or bulletin list
that an Admin/Ops could check frequently to find out if any of the tenants
on the NiFi service are using deprecated processors//controllers.

On Mon, May 1, 2017 at 8:24 AM, Rob Moran  wrote:

> I agree on treating these as we do restricted ones. I will work on some
> recommendations for a unique icon that the chooser dialog can display as
> well as how to best carry that over onto the canvas component.
>
> It would also be nice to suggest the alternative in the UI, for example
> when the user hovers the deprecated icon in the Add Processor dialog,
> specifically call out the alternative(s) by name.
>
> Rob
>
> On Apr 30, 2017 10:05 PM, "Andre"  wrote:
>
> All,
>
> Thank you for the feedback.
>
> I will extend the PR so that the required endpoints are in place but will
> leave the effective UI design to our UX experts.
>
> Hopefully I should be able to use the @Restricted as a basis for this work.
> Otherwise I will shout for some guidance.
>
> Cheers
>
>
> On Mon, May 1, 2017 at 11:23 AM, Joe Witt  wrote:
>
> > Agreed on both the icon being preferable as the vehicle to show this
> > for processors on the graph instead of color and also in emphasizing
> > this during processor selection.
> >
> > Thanks
> >
> > On Sun, Apr 30, 2017 at 7:21 PM, Matt Burgess 
> wrote:
> > > Visual indication on existing processors is a good thing, but IMO we'll
> > definitely need indication on the processor selection dialog, so the user
> > will know that we suggest they choose an alternative.
> > >
> > > Regards,
> > > Matt
> > >
> > >> On Apr 30, 2017, at 7:12 PM, Jeff  wrote:
> > >>
> > >> I think an icon on the processor would be best to denote an upcoming
> > >> deprecation.  That leaves all colors available to the flow design,
> since
> > >> any color may have specific meaning to any current flow out there.
> > >>
> > >>> On Sat, Apr 29, 2017 at 3:58 PM Juan Sequeiros 
> > wrote:
> > >>>
> > >>> In same train of though maybe make it default to some color other
> than
> > >>> white, instinct says red but that might be used to represent an
> > important
> > >>> processor. So suggest maybe default beige?
> > >>>
> > >>>
> > >>> On Sat, Apr 29, 2017 at 8:20 AM Andy LoPresto <
> > alopresto.apa...@gmail.com>
> > >>> wrote:
> > >>>
> >  I'd leave the graphics work/decisions to someone like Rob Moran, but
> I
> > >>> see
> >  it as similar to the restricted components -- an icon on the
> > processors
> > >>> on
> >  the canvas and the dialog to add components, with additional
> > explanation
> >  (and maybe a target release for full deprecation) in the help notes.
> > 
> >  Andy LoPresto
> >  alopre...@apache.org
> >  alopresto.apa...@gmail.com
> >  PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> > 
> > > On Apr 29, 2017, at 05:16, Andre  wrote:
> > >
> > > dev,
> > >
> > > In light of the some recent deprecations (ListenLumberjack) and
> some
> > > potential deprecation (PutJMS/GetJMS) I started some progress on
> how
> > to
> > > document the depreciation of NiFi components using annotations.
> > >
> > > In the absence of any better name I called the annotation
> > > @DeprecationWarning (so not overlap with Java's @Deprecated)
> > >
> > > The suggested modifications are part of PR#1718 and I am keen to
> hear
> >  your
> > > thoughts.
> > >
> > > While the documentation can be displayed as part of the usage,
> > however
> > >>> I
> > > would imagine the frequency a user browses to the usage pages will
> > >>> reduce
> > > as her/him becomes familiar with the components.
> > >
> > > In light of this it may be a good idea to use  the web UI to
> > highlight
> > >>> a
> > > processor has been deprecated.
> > >
> > > Would anyone suggest a way of doing so without disrupting the user
> > > experience?
> > >
> > > Cheers
> > 
> > >>>
> >
>



-- 
Joseph


Re: [CALL FOR ACTION] Peer reviews & testing of new code

2017-05-02 Thread Joseph Niemiec
Thanks for pointing out that *contributions *of Reviewing code are still
valid and that its not a *committer *only job!

On Mon, May 1, 2017 at 7:17 PM, Andre  wrote:

> Dev,
>
> Those following the git commit log probably noticed that there has been a
> lot of action by some of the developers behind some significant
> improvements in 1.2.0.
>
> As these developers get busy I would like to remind the rest of the list
> about importance of our contribution around code review and *testing* of
> [proposed] changes to the NiFi code base.
>
> So long history short. No matter if you have committer privilege to the
> NiFi git, you can still contribute to this great community.
>
> You can still contribute by providing extra testing of the introduced code
> (I am myself running SNAPSHOT in prod-like conditions with the new
> provenance WAL and it is so fast that I reckon you should be doing the
> same...).
>
>
> You can contribute by helping with the peer review process. [1]
>
>
> So if you have some cycles available and is keen be more active in our
> vibrant community, feel free to get involved!
>
>
>
> Cheers!
>
>
>
> [1] I admit that some of these PRs is daunting to the non-initiated (myself
> included), but trust me when I say that some are perfectly within reach of
> those that are familiar with NiFi and have also dealt with things like:
> HDFS, Azure Cloud, Websockets, etc etc etc. (To be honest, sometimes the
> contribution may be as simple as spell checking!!)
>



-- 
Joseph


Re: [VOTE] Release Apache NiFi 1.2.0 (RC2)

2017-05-08 Thread Joseph Niemiec
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

+1

* Verified Hashes
* Verified Signature
* LICENSE and NOTICE appear ok (but its my first time looking at these too)
* Build with - mvn -T 4C -Pcontrib-check clean install and passing
* OS: Ubuntu 14.04, Oracle JDK - jdk1.8.0_77-b03  [Linux edge1
4.4.0-75-generic #96~14.04.1-Ubuntu SMP Thu Apr 20 11:06:30 UTC 2017 x86_64
x86_64 x86_64 GNU/Linux]
* Ran Custom Sample Workflow - ListenUDP + ExecuteScript [Python] +
SplitContent + MergeContent + GZip + RenameFile +  FTP Upload
-BEGIN PGP SIGNATURE-
Version: Keybase OpenPGP v2.0.70
Comment: https://keybase.io/crypto

wsFcBAABCgAGBQJZEILFAAoJEJ+2VIrnunlacakP+wb1b3+I2mTuDMbdfb2E5z6p
RT9LgqMxHyTOGfdgHZrasOjbKdDeqJeb7q07s7cGBa2jo/ILrrEKdthfDUeXpTeA
MuoDaNyMxaFGZHkODeKxdFiZq/Cif4Q3eB2tqFMtx4R2YTmGCsVsbSEoWgVXXvxT
j5NPQx4DARsFyyTuwRH5M0H0gD03VioNRto3p2XlqPn4YdUovOSZQlclSb/fYm2z
7m3SVrK7kBM9WhxIGjJQG4x7QMGDRtQjPy4nPd1c0TuZszvc/LKeboAyRCU8pwXT
a5cIA8/PIKDd0VeiLKtccFJ0dKdWqSjT0yumoGEYrFUnlkhXKH4s2aBmxLRe362D
x043yiA22YYUMcUDb9c8deDKyNlHut9VgzdaNFZSlX4z805DavgRiZ4FeiflGRSH
3FenuBe36p6lR750oGYd4wKiiAhcm8U6cIx2RTSfITamNrlup4itehjFQU2Wod5R
M9nELlvrbsDxblOCdLBJlCH4a+/L3FvaUkxXeMHL3jamjxPj7WAHk0dTu1qrvHcO
OxDu11h8irWuh1GD/C0bWI4Xp2oacTB3Qvv9a4jY4UpPJ5eKE/1YXX45tiBV751x
G58JmtakGQgppil9OX/4XgnlK3AwQWRedddNg50KVcT/i1XKUWM4CeC1uHYQ0UlT
n30vmfIIgQlaBjO5DuFp
=RxKB
-END PGP SIGNATURE-



On Mon, May 8, 2017 at 2:12 AM, Andy LoPresto 
wrote:

> Joey,
>
> I am curious about the test failure. That is the test I refactored due to
> the POSIX attribute issue failing on Windows right before the RC. It passed
> when I built on Linux so if you can open a Jira or at least provide me with
> a stack trace and any steps to make it reproducible, I would appreciate it.
>
> Andy LoPresto
> alopre...@apache.org
> alopresto.apa...@gmail.com
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> > On May 7, 2017, at 22:17, Joey Frazee  wrote:
> >
> > +1 (non-binding)
> >
> > - Verified signature, checksums and commit id
> > - Successfully ran `mvn clean install -Pcontrib-check` on OS X 10.12.4
> with Oracle JDK 1.8.0_121 and Amazon Linux with OpenJDK 1.8.0_121
> > - Successfully ran `mvn clean install -DskipTests` on Docker
> maven:latest with OpenJDK 1.8.0_121 but CryptoUtilsTest.
> testShouldNotValidateUnreadableFileBasedKeyProvider always failed with
> !unreadableKeyProviderIsValid
> > - Built RPM with `mvn -T 4.0C clean install -Prpm,generateArchives
> -DskipTests` and tested RPM install
> > - Tested GenerateTableFetch and QueryDatabaseTable with MS SQL 2008
> [NIFI-3585], among other flows.
> >
> >> On May 5, 2017, at 9:07 PM, Bryan Bende  wrote:
> >>
> >> Hello,
> >>
> >> I am pleased to be calling this vote for the source release of Apache
> >> NiFi nifi-1.2.0 (RC2).
> >>
> >> The source zip, including signatures, digests, etc. can be found at:
> >> https://repository.apache.org/content/repositories/orgapachenifi-1104
> >>
> >> The Git tag is nifi-1.2.0-RC2
> >> The Git commit ID is 3a605af8e0ac024fb0ba67262d49dab2727b2576
> >> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=
> 3a605af8e0ac024fb0ba67262d49dab2727b2576
> >>
> >> Checksums of nifi-1.2.0-source-release.zip:
> >> MD5: 90e298a9e23a9dab65358daddd8b5990
> >> SHA1: e138941f576bdb1dff17df6674c19ffae3ef6719
> >> SHA256: c18398800c435dabff9f45ec55a450ca78c3c1aec222aa295c0e2057912f
> eeb6
> >>
> >> Release artifacts are signed with the following key:
> >> https://people.apache.org/keys/committer/bbende.asc
> >>
> >> KEYS file available here:
> >> https://dist.apache.org/repos/dist/release/nifi/KEYS
> >>
> >> 381 issues were closed/resolved for this release:
> >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> projectId=12316020&version=12338432
> >>
> >> Release note highlights can be found here:
> >> https://cwiki.apache.org/confluence/display/NIFI/
> Release+Notes#ReleaseNotes-Version1.2.0
> >>
> >> The vote will be open for 72 hours.
> >> Please download the release candidate and evaluate the necessary items
> >> including checking hashes, signatures, build
> >> from source, and test.  The please vote:
> >>
> >> [ ] +1 Release this package as nifi-1.2.0
> >> [ ] +0 no opinion
> >> [ ] -1 Do not release this package because because...
> >
>



-- 
Joseph


Re: [CANCEL][VOTE] Release Apache NiFi MiNiFI C++ 0.2.0 (RC1)

2017-05-08 Thread Joseph Niemiec
+0 (1 Test is failing for me)

* Sig Validated
* Hashes Validated
* Built on Ubuntu 14.04 [Used Ubuntu Package libleveldb1 & cmake 3.2
successfully: Linux edge1 4.4.0-75-generic #96~14.04.1-Ubuntu SMP Thu Apr
20 11:06:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux]
* Running tests I hit a segfault, checking to see if its specific my
environment or not. Working with Mark Parisi to identify root.



Test YAML Config 1
---
/data/ssd01/builds/minifi-020/nifi-minifi-cpp-0.2.0-source/libminifi/test/unit/YamlCongifurationTests.cpp:28
...
/data/ssd01/builds/minifi-020/nifi-minifi-cpp-0.2.0-source/libminifi/test/unit/YamlCongifurationTests.cpp:28:
FAILED:
due to a fatal error condition:
  SIGSEGV - Segmentation violation signal
===
test cases: 15 | 14 passed | 1 failed
assertions: 39 | 38 passed | 1 failed

On Sun, May 7, 2017 at 2:28 PM, Aldrin Piri  wrote:

> Looks like we have removed the problematic areas of the third party
> inclusions, updated the README and updated the tarball generation to avoid
> including generated doc files.  I will being working on an RC2 based off of
> commit hash fa913b4174e1c874ddb12e4901c48bd86a7f70e4.
>
> Thanks!
> Aldrin
>
> On Sat, May 6, 2017 at 10:41 AM, Aldrin Piri  wrote:
>
> > Yep, definitely seems like a good candidate to also include.  Will review
> > that as well and get incorporated if all looks good.  Thanks for bringing
> > to my attention.
> >
> > --aldrin
> >
> > On Sat, May 6, 2017 at 10:35 AM, Marc  wrote:
> >
> >> Aldrin,
> >>   I was preparing a response that we may wish to update the readme per
> >> https://github.com/apache/nifi-minifi-cpp/pull/92 . We don't include
> the
> >> full list of processors. Since the list is small it would be useful to
> >> keep
> >> it up to date at the moment. In the future we can prepare some auto
> >> documentation that we link to from the readme.
> >>
> >>
> >> On Sat, May 6, 2017 at 9:55 AM, Aldrin Piri  wrote:
> >>
> >> > All,
> >> >
> >> > After doing some review for the release process, it was seen that some
> >> > artifacts of our third party dependencies had unknown origins making
> it
> >> > hard to determine the full scope of licensing concerns.  To avoid any
> >> > issues, I am cancelling this vote such that they can be removed.
> >> >
> >> > I have created JIRA issue https://issues.apache.org/jira
> >> /browse/MINIFI-301 to
> >> > track this effort.
> >> >
> >> > Please look for an RC2 later today.
> >> >
> >> > Thanks and apologies,
> >> > Aldrin
> >> >
> >> >
> >> > On May 5, 2017, at 11:40, Aldrin Piri  wrote:
> >> >
> >> > Hello Apache NiFi Community,
> >> >
> >> > I am pleased to be calling this vote for the source release of Apache
> >> NiFi
> >> > MiNiFi C++, nifi-minifi-cpp-0.2.0.
> >> >
> >> > The source archive, signature, and digests can be located at:
> >> >
> >> > Source Archive:
> >> >https://dist.apache.org/repos/dist/dev/nifi/nifi-
> >> > minifi-cpp/0.2.0/nifi-minifi-cpp-0.2.0-source.tar.gz
> >> > GPG armored signature:
> >> >https://dist.apache.org/repos/dist/dev/nifi/nifi-
> >> > minifi-cpp/0.2.0/nifi-minifi-cpp-0.2.0-source.tar.gz.asc
> >> > Source MD5:
> >> >https://dist.apache.org/repos/dist/dev/nifi/nifi-
> >> > minifi-cpp/0.2.0/nifi-minifi-cpp-0.2.0-source.tar.gz.md5
> >> > Source SHA1:
> >> >https://dist.apache.org/repos/dist/dev/nifi/nifi-
> >> > minifi-cpp/0.2.0/nifi-minifi-cpp-0.2.0-source.tar.gz.sha1
> >> > Source SHA256:
> >> >https://dist.apache.org/repos/dist/dev/nifi/nifi-
> >> > minifi-cpp/0.2.0/nifi-minifi-cpp-0.2.0-source.tar.gz.sha256
> >> >
> >> > The Git tag is minifi-cpp-0.2.0-RC1
> >> > The Git commit hash is b8103f03998820449c24502fa61c224c2e39e098
> >> > * https://git-wip-us.apache.org/repos/asf?p=nifi-minifi-cpp.
> >> > git;a=commit;h=b8103f03998820449c24502fa61c224c2e39e098
> >> > * https://github.com/apache/nifi-minifi-cpp/commit/
> >> > b8103f03998820449c24502fa61c224c2e39e098
> >> >
> >> > Checksums of nifi-minifi-cpp-0.2.0-source.tar.gz:
> >> > MD5: 4aeb83102e95a3d9c3601d1771e67106
> >> > SHA1: f6501f87bde981e31430b13b635bee944d4b3ab3
> >> > SHA256: 8d97eb17c328637058040357523ca99984b4dcfce77d3b5f81128e1f165a
> >> f081
> >> >
> >> > Release artifacts are signed with the following key:
> >> > https://people.apache.org/keys/committer/aldrin
> >> >
> >> > KEYS file available here:
> >> > https://dist.apache.org/repos/dist/release/nifi/KEYS
> >> >
> >> > 63 issues were closed/resolved for this release:
> >> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> >> > version=12338790&projectId=12319921
> >> >
> >> > Release note highlights can be found here:
> >> > https://cwiki.apache.org/confluence/display/MINIFI/
> >> > Release+Notes#ReleaseNotes-Versioncpp-0.2.0
> >> >
> >> > The vote will be extended to acco

Re: [EXT] Re: NiFi Throughput and Slowness

2017-07-03 Thread Joseph Niemiec
Is there a reason your using a RAID-5 Array?

Write speed on RAID-5 is greatly influenced by the controller itself. We
typically do setups using RAID-10 today now for better read/write
performance. The 'parity' bits of RAID5 cause a number of re-writes &
re-reads (4 iops) before it writes out the parity bits to the correct disk.
AKA You get 4 OPS for every 1 write in RAID5, meaning take your IOPS/4 in
the very least.  Don't get me wrong the IOPs provided by the SSD are ample,
but is not something I see anymore in the wild new (RAID5.)

Besides the RAID implementation I would definitely look at what Joe has
recommended.

On Mon, Jul 3, 2017 at 2:30 PM, Joe Witt  wrote:

> Karthik
>
> It is really important to follow best practice configuration for placement
> of repos to your underlying storage.  Configured well you can have hundreds
> of MB/s sustained throughput per node.
>
> Also be sure to take advantage of the record reader / writer capability if
> appropriate for your flow.  Configured well hundreds you can achieve of
> thousands of records per second through a series of enrichments, SQL based
> queries, with transformation all with schema and format awareness while
> many other flows happen at once.
>
> Also, while the ram is awesome that you have the JVM might not be able to
> take advantage of that well in a garbage collection friendly manner.
> Consider dialing that way down to say 8GB.
>
> If you have split text in there be sure it isn't splitting 10s of thousands
> or more records at once.  You can do two phase splits and see much better
> behavior.
>
> With that hardware sustained performance can be extremely high.  I'd say do
> a few raid1 instead of raid5.  You can then partition the various
> repositories of nifi to minimize the use of the same physical device and
> maximize throughput and response time.   You'll also prob need 10Gb
> NICs/network.
>
> And clustering is a powerful feature.  I'd avoid doing that until you have
> the right fundamentals at play in a single node and and see both the
> sustained throughout and transaction rate you'd expect.
>
> Thanks
> Joe
>
>
>
>
>
> On Jul 3, 2017 1:18 PM, "Karthik Kothareddy (karthikk) [CONT - Type 2]" <
> karth...@micron.com> wrote:
>
> Rick,
>
> Thanks a lot for the suggestion, clustering is something that even I was
> thinking of for a long time. Just wanted to see if anyone in the community
> have similar problems and solutions they found.
>
> -Karthik
>
> -Original Message-
> From: Richard St. John [mailto:rstjoh...@gmail.com]
> Sent: Monday, July 03, 2017 10:54 AM
> To: dev@nifi.apache.org; dev@nifi.apache.org
> Subject: [EXT] Re: NiFi Throughput and Slowness
>
> Hi there,
>
> In the beginning of our NiFi adoption, we faced similar issues. For us, we
> clustered NiFi, limited the number of concurrent tasks for each processor
> and added more logical partitions for content and provenance repositories.
> Now, we easily processor million of flow files per minute on a 5-node
> cluster with hundreds of processors in the data flow pipeline. When we need
> to ingest more data or process it faster, we simply add more nodes.
>
> First and foremost, clustering NiFi allows horizontal scaling: a must. It
> seems counterintuitive, but limiting the number of concurrent tasks was a
> major performance improvement. Doing so keeps the flow "balanced",
> preventing hotspots within the flow pipeline.
>
> I hope this helps
>
> Rick.
>
> --
> Richard St. John, PhD
> Asymmetrik
> 141 National Business Pkwy, Suite 110
> Annapolis Junction, MD 20701
>
> On Jul 3, 2017, 12:53 PM -0400, Karthik Kothareddy (karthikk) [CONT - Type
> 2] , wrote:
> > All,
> >
> > I am currently using NiFi 1.2.0 on a Linux (RHEL) machine. I am using a
> single instance without any clustering. My machine has ~800GB of RAM and
> 2.5 TB of disk space (SSD’s with RAID5). I have set my Java heap space
> values to below in “bootstrap.conf” file
> >
> > # JVM memory settings
> > java.arg.2=-Xms40960m
> > java.arg.3=-Xmx81920m
> >
> > # Some custom Configurations
> > java.arg.7=-XX:ReservedCodeCacheSize=1024m
> > java.arg.8=-XX:CodeCacheMinimumFreeSpace=10m
> > java.arg.9=-XX:+UseCodeCacheFlushing
> >
> > Now, the problem that I am facing when I am stress testing this instance
> is whenever the Read/Write of Data feeds reach the limit of 5GB (at least
> that’s what I observed) the whole instance is running super slow meaning
> the flowfiles are moving very slow in the queues. It is heavily affecting
> the other Processor groups as well which are very simple flows. I tied to
> read the system diagnostics at that point and see that all the usage is
> below 20% including heap Usage, flowFile and content repository usage. I
> tried to capture the status history of the Process Group at that particular
> point and below are some results.
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > From the above images it is obvious that the process group is working on
> lot of IO at that point. Is there a way

Re: NIFI Expression Language to Evaluate custom function

2017-07-18 Thread Joseph Niemiec
I believe what your asking for is possible, I am not very familiar with
Spring though. Check out this commit for the base64encode/decode, lemme
know if it helps you out.

https://github.com/apache/nifi/commit/6f1af31ff28f60d0eddbee5dafe909bc66cc9c71



On Tue, Jul 18, 2017 at 7:04 AM, hemantvsn  wrote:

> Can NIFI expression language be used to evaluate custom functions.
>
> For instance, we need to invoke the function,
> new org.bson.types.ObjectId().toHexString()
>
> and assign its output to our attribute, is it possible solely using EL?
> Im drawing its parallel with SPEL (Spring expression language)
>
> http://docs.spring.io/autorepo/docs/spring-framework/4.0.0.RC2/spring-
> framework-reference/html/expressions.html
>
> Eg: Class dateClass =
> parser.parseExpression("T(java.util.Date)").getValue(Class.class);
>
> Class stringClass =
> parser.parseExpression("T(String)").getValue(Class.class);
>
>
>
>
>
> --
> View this message in context: http://apache-nifi-developer-
> list.39713.n7.nabble.com/NIFI-Expression-Language-to-
> Evaluate-custom-function-tp16451.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>



-- 
Joseph


Re: [DISCUSS] Increasing durability in MiNiFi C++

2017-08-01 Thread Joseph Niemiec
I feel that MINIFI-356 is pretty key in all things, when I think of jagged
edge use cases that are missing connectivity for days but have large mass
storage devices this feels really limiting.  When I consider the variety of
devices I have tested with thus far most of them only have a single storage
media mount. RasPi's right now seem to be the de-typical entry IoT and most
are only using the single media mount this to me represents an area where
even operating in degraded mode won't help us as the OS will fail on its
own eventually without its disk.

With that said is it more valuable to use the storage media we have
initially then it is to find a way to run without it?


No doubt there are other scenarios where this is very useful and I see more
of them initially in the 'non-jagged' space. For example a factory line PC
within the Enterprise network is always connected, it may never experience
backpressure soley because it can send as fast as it collects the data. If
we assume that the OS disks and Repo disks are not the same, and the repo
did fail there would be value in continuing to operate collecting and
sending data, but for all intents we dont care about backpressure here
becuase we can still send it as fast as its collected.

~~Kevins' Response's

2. Logging and readme documentation will be important to assist
troubleshooting / debugging. If an agent is configured to use a persistent
repository, and it has degraded to a volatile repository, that could be
really confusing to a novice user/admin who is trying to figure out how the
agent is working. Therefore we need to make sure changes to agent behavior
that occur as part of continuing operations are logged at some level.

I would also expect initially its default off, and has to be manually
enabled.


3. Testing

Just intally thinking I can re-use a RasPi but attach an ESATA, a hard
failure of removing the drive itself, or unmounting it at the OS level may
do this. While leaving the OS drive (SD card) still plugged in.



On Tue, Aug 1, 2017 at 9:59 AM, Marc  wrote:

> Good Morning,
>
>   I've begun capturing some details in a ticket for durability and
> reliability of MiNiFi C++ clients [1]. The scope of this ticket is
> continuing operations despite failure within specific components. There is
> a linked ticket [2] attempts to address some of the concerns brought up in
> MINIFI-356, focusing no memory usage.
>
>   The spirit of the ticket was meant to capture conditions of known
> failure; however, given that more discussion has blossomed, I'd like to
> assess the experience of the mailing list. Continuing operations in any
> environment is difficult, particularly one in which we likely have little
> to no control. Simply gathering information to know when a failure is
> occurring is a major part of the battle. According to the tickets, there
> needs to be some discussion of how we classify failure.
>
>   The ticket addressed the low hanging fruit, but there are certainly more
> conditions of failure. If a disk switches to read/write mode, disks becomes
> full and/or out of inode entries etc, we know a complete failure occurred
> and thus can switch our type of write activity to use a volatile repo. I
> recognize that partial failures may occur, but how do we classify these?
> Should we classify these at all or would this be venturing into a rabbit
> hole?
>
>For memory we can likely throttle queue sizes as needed. For networking
> and other components we could likely find other measures of failure. The
> goal, no matter the component, is to continue operations without human
> intervention -- with the hope that the configuration makes the bounds of
> the client obvious.
>
>My gut reaction is to separate partial failure as the low hanging fruit
> of complete failure is much easier to address, but would love to hear the
> reaction of this list. Further, any input on the types of failures to
> address would be appreciated. Look forward to any and all responses.
>
>   Best Regards,
>   Marc
>
> [1] https://issues.apache.org/jira/browse/MINIFI-356
> [2] https://issues.apache.org/jira/browse/MINIFI-360
>



-- 
Joseph


Re: [DISCUSS] Apache NiFi distribution has grown too large

2018-01-12 Thread Joseph Niemiec
just a random thought.

Drop In Lib packs... All the Hadoop ones in one package for example that
can be added to a slim Nifi install. Another may be for Cloud, or Database
Interactions, Integration (JMS, FTP, etc) of course defining these groups
would be the tricky part... Or perhaps some type of installer which allows
you to elect which packages to download to add to the slim install?


On Fri, Jan 12, 2018 at 3:10 PM, Joe Witt  wrote:

> Team,
>
> The NiFi convenience binary (tar.gz/zip) size has grown to 1.1GB now
> in the latest release.  Apache infra expanded it to 1.6GB allowance
> for us but has stated this is the last time.
> https://issues.apache.org/jira/browse/INFRA-15816
>
> We need consider:
> 1) removing old nars/less commonly used nars/or particularly massive
> nars from the assembly we distribute by default.  Folks can still use
> these things if they want just not from our convenience binary
> 2) collapsing nars with highly repeating deps
> 3) Getting the extension registry baked into the Flow Registry then
> moving to separate releases for extension bundles.  The main release
> then would be just the NiFi framework.
>
> Any other ideas ?
>
> I'll plan to start identifying candiates for removal soon.
>
> Thanks
> Joe
>



-- 
Joseph


Docker NiFi Container superuser password

2018-03-28 Thread Joseph Niemiec
Hi everyone.

Looking to change the nameservers in the default docker image
apache/nifi:latest but am unsure of the password...

nifi@39dbe311249e:/opt/nifi/nifi-1.5.0$ su
Password:  

Re: Docker NiFi Container superuser password

2018-03-28 Thread Joseph Niemiec
please ignore, i managed to do this by telling docker to login as root and
not the default...

*docker exec -i -t --user root 39dbe311249e bash *

On Wed, Mar 28, 2018 at 2:06 PM, Joseph Niemiec 
wrote:

> Hi everyone.
>
> Looking to change the nameservers in the default docker image
> apache/nifi:latest but am unsure of the password...
>
> nifi@39dbe311249e:/opt/nifi/nifi-1.5.0$ su
> Password:   su: Authentication failure
>
> Is this something that is publicly available?
>
>
> --
> Joseph
>



-- 
Joseph