Re: [DRAFT][REPORT] Apache NiFi Board Report Oct 2017

[Joe Witt]

thanks all very much for the prompt responses!  Submitted with drew's edits.

joe

On Mon, Oct 9, 2017 at 10:11 PM, James Wing  wrote:
> Thanks, Joe. It looks good to me, too.
>
> On Mon, Oct 9, 2017 at 7:12 AM, Joe Witt  wrote:
>
>> Team,
>>
>> Please see draft board report.  Am looking for feedback/edits quickly
>> as this needs to be submitted by Wed.  I'll probably send it in
>> tonight though as I might not have time over the next couple of days.
>>
>> Thanks
>> Joe
>>
>> **==**==**==**==**
>>
>> ## Description:
>>  - Apache NiFi is an easy to use, powerful, and reliable system to
>>process and distribute data.
>>  - Apache NiFi MiNiFi is an edge data collection agent built to seamlessly
>>integrate with and leverage the command and control of NiFi. There are
>>both Java and C++ implementations.
>>  - Apache NiFi Registry is a centralized registry for key configuration
>> items
>>including flow versions, assets, and extensions for Apache NiFi
>>and Apache MiNiFi.
>>  - Apache NiFi Nar Maven Plugin is a release artifact used for supporting
>>the NiFi classloader isolation model.
>>
>> ## Issues:
>>  - There are no issues requiring board attention at this time.
>>
>> ## Activity:
>>  - Released Apache NiFi 1.4.0 which is another very heavy feature,
>> stability,
>>and bug fix release.  It includes improvements on the web UI coming
>> from new
>>contributors which is a great sign.
>>  - The Apache NiFi downloads have been quite large and growing for some
>> time.
>>The groundwork to solve this is the Apache NiFi Registry effort which is
>>coming along nicely and which the foundational elements of NiFi to
>> integrate
>>with it are present.  We're probably looking at two more releases for
>> that
>>to land and for which we can drop the NiFi build size down.
>>  - We continue to hover around the 100 outstanding pull requests number.
>> What
>>is encouraging is we've seen an uptick in contributions both in code and
>>review feedback from new community members.
>>  - The Apache NiFi MiNiFi C++ agent has rapidly matured over the past
>> couple of
>>months and is serving as a new source for contributions and
>> contributors.
>>
>> ## Health report:
>>  - As stated in our past report, health of the community is strong and
>>indicators of strength continue trending in the right direction. Mailing
>>list and JIRA activity is strong. ASF Hipchat is serving as an on-ramp
>> for
>>new users to our mailing list and JIRA systems. We continue to see new
>>users and contributors.
>>  - The PMC and committer ranks did not grow during the reporting period.
>> This
>>is a reflection of the PMC not focusing enough on the topic though as
>> we're
>>seeing a very strong pipeline to both both committer and PMC. Expect
>> both
>>new committers and PMC members during our next reporting period.
>>
>> ## PMC changes:
>>
>>  - Currently 22 PMC members.
>>  - Last PMC member added Fri May 26 2017.
>>
>> ## Committer base changes:
>>
>>  - Currently 34 committers.
>>  - Last committer added Thu Jun 01 2017.
>>
>> ## Releases:
>>
>>  - Apache NiFi 1.4.0 was released Oct 1 2017.
>>
>> ## Mailing list activity:
>>
>>  - Activity on the mailing lists remains extremely high with a mixture
>>of new users, contributors, and deeper more experienced users and
>>contributors sparking discussion and questions and filing bugs or
>>new features.
>>
>>  - us...@nifi.apache.org:
>> - 553 subscribers (up 34 in the last 3 months):
>> - 748 emails sent to list (862 in previous quarter)
>>
>>  - dev@nifi.apache.org:
>> - 383 subscribers (up 12 in the last 3 months):
>> - 604 emails sent to list (947 in previous quarter)
>>
>>  - iss...@nifi.apache.org:
>> - 46 subscribers (up 4 in the last 3 months):
>> - 5775 emails sent to list (7457 in previous quarter)
>>
>>
>> ## JIRA activity:
>>
>>  - 335 JIRA tickets created in the last 3 months
>>  - 263 JIRA tickets closed/resolved in the last 3 months
>>

Re: [DRAFT][REPORT] Apache NiFi Board Report Oct 2017

[James Wing]

Thanks, Joe. It looks good to me, too.

On Mon, Oct 9, 2017 at 7:12 AM, Joe Witt  wrote:

> Team,
>
> Please see draft board report.  Am looking for feedback/edits quickly
> as this needs to be submitted by Wed.  I'll probably send it in
> tonight though as I might not have time over the next couple of days.
>
> Thanks
> Joe
>
> **==**==**==**==**
>
> ## Description:
>  - Apache NiFi is an easy to use, powerful, and reliable system to
>process and distribute data.
>  - Apache NiFi MiNiFi is an edge data collection agent built to seamlessly
>integrate with and leverage the command and control of NiFi. There are
>both Java and C++ implementations.
>  - Apache NiFi Registry is a centralized registry for key configuration
> items
>including flow versions, assets, and extensions for Apache NiFi
>and Apache MiNiFi.
>  - Apache NiFi Nar Maven Plugin is a release artifact used for supporting
>the NiFi classloader isolation model.
>
> ## Issues:
>  - There are no issues requiring board attention at this time.
>
> ## Activity:
>  - Released Apache NiFi 1.4.0 which is another very heavy feature,
> stability,
>and bug fix release.  It includes improvements on the web UI coming
> from new
>contributors which is a great sign.
>  - The Apache NiFi downloads have been quite large and growing for some
> time.
>The groundwork to solve this is the Apache NiFi Registry effort which is
>coming along nicely and which the foundational elements of NiFi to
> integrate
>with it are present.  We're probably looking at two more releases for
> that
>to land and for which we can drop the NiFi build size down.
>  - We continue to hover around the 100 outstanding pull requests number.
> What
>is encouraging is we've seen an uptick in contributions both in code and
>review feedback from new community members.
>  - The Apache NiFi MiNiFi C++ agent has rapidly matured over the past
> couple of
>months and is serving as a new source for contributions and
> contributors.
>
> ## Health report:
>  - As stated in our past report, health of the community is strong and
>indicators of strength continue trending in the right direction. Mailing
>list and JIRA activity is strong. ASF Hipchat is serving as an on-ramp
> for
>new users to our mailing list and JIRA systems. We continue to see new
>users and contributors.
>  - The PMC and committer ranks did not grow during the reporting period.
> This
>is a reflection of the PMC not focusing enough on the topic though as
> we're
>seeing a very strong pipeline to both both committer and PMC. Expect
> both
>new committers and PMC members during our next reporting period.
>
> ## PMC changes:
>
>  - Currently 22 PMC members.
>  - Last PMC member added Fri May 26 2017.
>
> ## Committer base changes:
>
>  - Currently 34 committers.
>  - Last committer added Thu Jun 01 2017.
>
> ## Releases:
>
>  - Apache NiFi 1.4.0 was released Oct 1 2017.
>
> ## Mailing list activity:
>
>  - Activity on the mailing lists remains extremely high with a mixture
>of new users, contributors, and deeper more experienced users and
>contributors sparking discussion and questions and filing bugs or
>new features.
>
>  - us...@nifi.apache.org:
> - 553 subscribers (up 34 in the last 3 months):
> - 748 emails sent to list (862 in previous quarter)
>
>  - dev@nifi.apache.org:
> - 383 subscribers (up 12 in the last 3 months):
> - 604 emails sent to list (947 in previous quarter)
>
>  - iss...@nifi.apache.org:
> - 46 subscribers (up 4 in the last 3 months):
> - 5775 emails sent to list (7457 in previous quarter)
>
>
> ## JIRA activity:
>
>  - 335 JIRA tickets created in the last 3 months
>  - 263 JIRA tickets closed/resolved in the last 3 months
>

Re: [EXT] Re: Funnel Queue Slowness

[Joe Witt]

Peter,

I see your point that it feels not natural or at least surprising.
There are two challenges I see with what you propose.  One is user
oriented and the other is technical.

The user oriented one is that penalized objects are penalized as a
function of the thing that last operated on them.  The further away we
let the data get the harder it would be to reason over why they were
penalized in the first place.

The technical one is that once something is penalized and placed into
the queue there is prioritization and polling logic that kicks in as a
factor.  I'm not sure how we'd tweak it for that to be ok in some
cases and in others not.  Perhaps we could just make funnels truly a
pass-through and when calculating the queue we're storing on figure
out the first non-funnel queue provided there is no cloning/branching
we'd have to account for.  But even then it brings us back to the
previous point which is the user challenge of knowing what thing
penalized objects in queue in the first place.

Alternatively, we should review whether it is obvious enough (or at
all) that items within a queue at a given moment in time are
penalized.  I've worked with NiFi for a very long time and i'll be
honest and state I've forgotten that penalization was a thing more
than a few times too.

What do you think?

Thanks

On Mon, Oct 9, 2017 at 9:01 PM, Peter Wicks (pwicks)  wrote:
> Bryan,
>
> Yes, it was the penalty causing the issue. This feels like weird behavior for 
> Funnel’s, and I’m not sure if it makes sense for penalties to work this way.
>
> Would it make more sense if penalties were generally kept as is, but not 
> applied at Funnel’s, then the penalty would kick back in at the first 
> non-funnel queue?
>
> Thanks,
>   Peter
>
> From: Bryan Bende [mailto:bbe...@gmail.com]
> Sent: Monday, October 09, 2017 7:33 PM
> To: dev@nifi.apache.org
> Subject: [EXT] Re: Funnel Queue Slowness
>
> Peter,
>
> The images didn’t come across for me, but since you mentioned that a failure 
> queue is involved, is it possible all the flow files going to failure are 
> being penalized which would cause them to not be processed immediately?
>
> -Bryan
>
>
> On Oct 8, 2017, at 10:49 PM, Peter Wicks (pwicks) 
> > wrote:
>
> I’ve been running into an issue on 1.4.0 where my Funnel sometimes runs slow. 
> I haven’t been able to create a nice reproducible test case to pass on.
> What I’m seeing is that my failure queue on the right will start to fill up, 
> even though there is plenty of room for them in the next queue. You can see 
> that the Tasks/Time is fairly low, only 24 in the last 5 minutes (first 
> image), so it’s not that the FlowFile’s are moving so fast that they just 
> appear to be in queue.
>
> If I stop the downstream processor the files slowly trickle through the 
> funnel into the next queue slowly. I had an Oldest FlowFile First prioritizer 
> on the downstream queue. I tried removing it but there was no change in 
> behavior.
> One time where I saw this behavior in the past was when my NiFi instance was 
> thread starved, but there are plenty of threads available on the instance and 
> all other processors are running fine. I also don’t understand why it 
> trickles the FlowFile’s in, from what I’ve seen in the code Funnel grabs 
> large batches at one time…
>
> Thoughts?
>
> (Sometimes my images don’t make it, let me know if that happens.)
> [cid:image002.png@01D340EC.543FE750] [cid:image004.png@01D340EC.543FE750]
>

RE: [EXT] Re: Funnel Queue Slowness

[Peter Wicks (pwicks)]

Bryan,

Yes, it was the penalty causing the issue. This feels like weird behavior for 
Funnel’s, and I’m not sure if it makes sense for penalties to work this way.

Would it make more sense if penalties were generally kept as is, but not 
applied at Funnel’s, then the penalty would kick back in at the first 
non-funnel queue?

Thanks,
  Peter

From: Bryan Bende [mailto:bbe...@gmail.com]
Sent: Monday, October 09, 2017 7:33 PM
To: dev@nifi.apache.org
Subject: [EXT] Re: Funnel Queue Slowness

Peter,

The images didn’t come across for me, but since you mentioned that a failure 
queue is involved, is it possible all the flow files going to failure are being 
penalized which would cause them to not be processed immediately?

-Bryan


On Oct 8, 2017, at 10:49 PM, Peter Wicks (pwicks) 
> wrote:

I’ve been running into an issue on 1.4.0 where my Funnel sometimes runs slow. I 
haven’t been able to create a nice reproducible test case to pass on.
What I’m seeing is that my failure queue on the right will start to fill up, 
even though there is plenty of room for them in the next queue. You can see 
that the Tasks/Time is fairly low, only 24 in the last 5 minutes (first image), 
so it’s not that the FlowFile’s are moving so fast that they just appear to be 
in queue.

If I stop the downstream processor the files slowly trickle through the funnel 
into the next queue slowly. I had an Oldest FlowFile First prioritizer on the 
downstream queue. I tried removing it but there was no change in behavior.
One time where I saw this behavior in the past was when my NiFi instance was 
thread starved, but there are plenty of threads available on the instance and 
all other processors are running fine. I also don’t understand why it trickles 
the FlowFile’s in, from what I’ve seen in the code Funnel grabs large batches 
at one time…

Thoughts?

(Sometimes my images don’t make it, let me know if that happens.)
[cid:image002.png@01D340EC.543FE750] [cid:image004.png@01D340EC.543FE750]

[ANNOUNCE] Apache NiFi CVE-2017-12623

[Andy LoPresto]

Apache NiFi PMC would like to announce the discovery and resolution of 
CVE-2017-12623. This issue has been resolved and a new version of the Apache 
NiFi project was released in accordance with the Apache Release Process.

Apache NiFi is an easy to use, powerful, and reliable system to process and 
distribute data. It supports powerful and scalable directed graphs of data 
routing, transformation, and system mediation logic.

Fixed in Apache NiFi 1.4.0 (Released: October 2, 2017)

CVE-2017-12623: Apache NiFi XXE issue when loading template

Severity: Medium

Versions Affected:

Apache NiFi 1.0.0 - 1.3.0

Description: An authorized user could upload a template which contained 
malicious code and accessed sensitive files via an XML External Entity (XXE) 
attack.

Mitigation: The fix to properly handle XML External Entities was applied on the 
Apache NiFi 1.4.0 release. Users running a prior 1.x release should upgrade to 
the appropriate release.

Credit: This issue was discovered by Paweł Gocyla.

For more information: https://nifi.apache.org/security.html

Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: [DRAFT][REPORT] Apache NiFi Board Report Oct 2017

[Yolanda Davis]

LGTM as well with suggested update.

-yolanda

On Mon, Oct 9, 2017 at 11:03 AM, Rob Moran  wrote:

> Joe, looks good with the couple changes Drew mentioned--thanks.
>
> On Mon, Oct 9, 2017 at 10:30 AM Andrew Lim 
> wrote:
>
> > Noticed the following:
> >
> > -In Health Report section: "seeing a very strong pipeline to both both
> > committer and PMC” (remove extra both)
> > -"Apache NiFi 1.4.0 was released Oct 1 2017” (Change? Release Notes says
> > Oct 2)
> >
> > Otherwise, looks great!
> >
> > -Drew
> >
> > > On Oct 9, 2017, at 10:23 AM, Marc  wrote:
> > >
> > > Joe
> > >
> > >  I think this looks great.
> > >
> > >  Thanks for putting this together,
> > >
> > >
> > >
> > > On Mon, Oct 9, 2017 at 10:12 AM, Joe Witt  wrote:
> > >
> > >> Team,
> > >>
> > >> Please see draft board report.  Am looking for feedback/edits quickly
> > >> as this needs to be submitted by Wed.  I'll probably send it in
> > >> tonight though as I might not have time over the next couple of days.
> > >>
> > >> Thanks
> > >> Joe
> > >>
> > >> **==**==**==**==**
> > >>
> > >> ## Description:
> > >> - Apache NiFi is an easy to use, powerful, and reliable system to
> > >>   process and distribute data.
> > >> - Apache NiFi MiNiFi is an edge data collection agent built to
> > seamlessly
> > >>   integrate with and leverage the command and control of NiFi. There
> are
> > >>   both Java and C++ implementations.
> > >> - Apache NiFi Registry is a centralized registry for key configuration
> > >> items
> > >>   including flow versions, assets, and extensions for Apache NiFi
> > >>   and Apache MiNiFi.
> > >> - Apache NiFi Nar Maven Plugin is a release artifact used for
> supporting
> > >>   the NiFi classloader isolation model.
> > >>
> > >> ## Issues:
> > >> - There are no issues requiring board attention at this time.
> > >>
> > >> ## Activity:
> > >> - Released Apache NiFi 1.4.0 which is another very heavy feature,
> > >> stability,
> > >>   and bug fix release.  It includes improvements on the web UI coming
> > >> from new
> > >>   contributors which is a great sign.
> > >> - The Apache NiFi downloads have been quite large and growing for some
> > >> time.
> > >>   The groundwork to solve this is the Apache NiFi Registry effort
> which
> > is
> > >>   coming along nicely and which the foundational elements of NiFi to
> > >> integrate
> > >>   with it are present.  We're probably looking at two more releases
> for
> > >> that
> > >>   to land and for which we can drop the NiFi build size down.
> > >> - We continue to hover around the 100 outstanding pull requests
> number.
> > >> What
> > >>   is encouraging is we've seen an uptick in contributions both in code
> > and
> > >>   review feedback from new community members.
> > >> - The Apache NiFi MiNiFi C++ agent has rapidly matured over the past
> > >> couple of
> > >>   months and is serving as a new source for contributions and
> > >> contributors.
> > >>
> > >> ## Health report:
> > >> - As stated in our past report, health of the community is strong and
> > >>   indicators of strength continue trending in the right direction.
> > Mailing
> > >>   list and JIRA activity is strong. ASF Hipchat is serving as an
> on-ramp
> > >> for
> > >>   new users to our mailing list and JIRA systems. We continue to see
> new
> > >>   users and contributors.
> > >> - The PMC and committer ranks did not grow during the reporting
> period.
> > >> This
> > >>   is a reflection of the PMC not focusing enough on the topic though
> as
> > >> we're
> > >>   seeing a very strong pipeline to both both committer and PMC. Expect
> > >> both
> > >>   new committers and PMC members during our next reporting period.
> > >>
> > >> ## PMC changes:
> > >>
> > >> - Currently 22 PMC members.
> > >> - Last PMC member added Fri May 26 2017.
> > >>
> > >> ## Committer base changes:
> > >>
> > >> - Currently 34 committers.
> > >> - Last committer added Thu Jun 01 2017.
> > >>
> > >> ## Releases:
> > >>
> > >> - Apache NiFi 1.4.0 was released Oct 1 2017.
> > >>
> > >> ## Mailing list activity:
> > >>
> > >> - Activity on the mailing lists remains extremely high with a mixture
> > >>   of new users, contributors, and deeper more experienced users and
> > >>   contributors sparking discussion and questions and filing bugs or
> > >>   new features.
> > >>
> > >> - us...@nifi.apache.org:
> > >>- 553 subscribers (up 34 in the last 3 months):
> > >>- 748 emails sent to list (862 in previous quarter)
> > >>
> > >> - dev@nifi.apache.org:
> > >>- 383 subscribers (up 12 in the last 3 months):
> > >>- 604 emails sent to list (947 in previous quarter)
> > >>
> > >> - iss...@nifi.apache.org:
> > >>- 46 subscribers (up 4 in the last 3 months):
> > >>- 5775 emails sent to list (7457 in previous quarter)
> > >>
> > >>
> > >> ## JIRA activity:
> > >>
> > >> - 335 JIRA tickets created in the last 3 months
> > >> - 263 JIRA tickets 

Re: FuzzyHashContent/CompareFuzzyHash processor

[Andy LoPresto]

You need to extract the relevant fields and either modify the flowfile content 
inline (losing the other data) or create a new flowfile (you can still retain 
the complete content in the “original” flowfile) and pass the flowfile with 
only the content you want to perform the hash on to the FuzzyHashContent 
processor.

For the data you have provided (I’m assuming this is a single line of values, 
rather than the structure and there exist many lines), you could use a 
ReplaceText processor to drop unrelated columns. If you have multiple rows in 
the flowfile content, you can use a CSVRecordReader/ScriptedReader and 
CSVRecordSetWriter/ScriptedRecordSetWriter in conjunction with an UpdateRecord 
processor to reduce the content down to just the relevant fields, and then use 
a SplitRecord processor to generate individual flowfiles from each line, and 
pass all of them to FuzzyHashContent.


Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Oct 9, 2017, at 4:19 AM, shankhamajumdar  
> wrote:
> 
> Hi Andy,
> 
> Thanks for the reply. But I am still not able to solve my use case. For
> example
> 
> I have a data file in the below structure.
> 
> Col1  Col2  Col3  Col4  Col5
> 
> Test1Test2 Test3 Test4 Test5
> 
> I want to do a fuzzy matching on Col2 and Col3 and generate an output file.
> 
> I am using getFile and FuzzyHashContent processor but not able to design the
> flow. Need your help on this.
> 
> Regards,
> Shankha
> 
> 
> 
> 
> 
> 
> --
> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: [DRAFT][REPORT] Apache NiFi Board Report Oct 2017

[Andrew Psaltis]

Joe -- LGTM

On Mon, Oct 9, 2017 at 11:03 AM, Rob Moran  wrote:

> Joe, looks good with the couple changes Drew mentioned--thanks.
>
> On Mon, Oct 9, 2017 at 10:30 AM Andrew Lim 
> wrote:
>
> > Noticed the following:
> >
> > -In Health Report section: "seeing a very strong pipeline to both both
> > committer and PMC” (remove extra both)
> > -"Apache NiFi 1.4.0 was released Oct 1 2017” (Change? Release Notes says
> > Oct 2)
> >
> > Otherwise, looks great!
> >
> > -Drew
> >
> > > On Oct 9, 2017, at 10:23 AM, Marc  wrote:
> > >
> > > Joe
> > >
> > >  I think this looks great.
> > >
> > >  Thanks for putting this together,
> > >
> > >
> > >
> > > On Mon, Oct 9, 2017 at 10:12 AM, Joe Witt  wrote:
> > >
> > >> Team,
> > >>
> > >> Please see draft board report.  Am looking for feedback/edits quickly
> > >> as this needs to be submitted by Wed.  I'll probably send it in
> > >> tonight though as I might not have time over the next couple of days.
> > >>
> > >> Thanks
> > >> Joe
> > >>
> > >> **==**==**==**==**
> > >>
> > >> ## Description:
> > >> - Apache NiFi is an easy to use, powerful, and reliable system to
> > >>   process and distribute data.
> > >> - Apache NiFi MiNiFi is an edge data collection agent built to
> > seamlessly
> > >>   integrate with and leverage the command and control of NiFi. There
> are
> > >>   both Java and C++ implementations.
> > >> - Apache NiFi Registry is a centralized registry for key configuration
> > >> items
> > >>   including flow versions, assets, and extensions for Apache NiFi
> > >>   and Apache MiNiFi.
> > >> - Apache NiFi Nar Maven Plugin is a release artifact used for
> supporting
> > >>   the NiFi classloader isolation model.
> > >>
> > >> ## Issues:
> > >> - There are no issues requiring board attention at this time.
> > >>
> > >> ## Activity:
> > >> - Released Apache NiFi 1.4.0 which is another very heavy feature,
> > >> stability,
> > >>   and bug fix release.  It includes improvements on the web UI coming
> > >> from new
> > >>   contributors which is a great sign.
> > >> - The Apache NiFi downloads have been quite large and growing for some
> > >> time.
> > >>   The groundwork to solve this is the Apache NiFi Registry effort
> which
> > is
> > >>   coming along nicely and which the foundational elements of NiFi to
> > >> integrate
> > >>   with it are present.  We're probably looking at two more releases
> for
> > >> that
> > >>   to land and for which we can drop the NiFi build size down.
> > >> - We continue to hover around the 100 outstanding pull requests
> number.
> > >> What
> > >>   is encouraging is we've seen an uptick in contributions both in code
> > and
> > >>   review feedback from new community members.
> > >> - The Apache NiFi MiNiFi C++ agent has rapidly matured over the past
> > >> couple of
> > >>   months and is serving as a new source for contributions and
> > >> contributors.
> > >>
> > >> ## Health report:
> > >> - As stated in our past report, health of the community is strong and
> > >>   indicators of strength continue trending in the right direction.
> > Mailing
> > >>   list and JIRA activity is strong. ASF Hipchat is serving as an
> on-ramp
> > >> for
> > >>   new users to our mailing list and JIRA systems. We continue to see
> new
> > >>   users and contributors.
> > >> - The PMC and committer ranks did not grow during the reporting
> period.
> > >> This
> > >>   is a reflection of the PMC not focusing enough on the topic though
> as
> > >> we're
> > >>   seeing a very strong pipeline to both both committer and PMC. Expect
> > >> both
> > >>   new committers and PMC members during our next reporting period.
> > >>
> > >> ## PMC changes:
> > >>
> > >> - Currently 22 PMC members.
> > >> - Last PMC member added Fri May 26 2017.
> > >>
> > >> ## Committer base changes:
> > >>
> > >> - Currently 34 committers.
> > >> - Last committer added Thu Jun 01 2017.
> > >>
> > >> ## Releases:
> > >>
> > >> - Apache NiFi 1.4.0 was released Oct 1 2017.
> > >>
> > >> ## Mailing list activity:
> > >>
> > >> - Activity on the mailing lists remains extremely high with a mixture
> > >>   of new users, contributors, and deeper more experienced users and
> > >>   contributors sparking discussion and questions and filing bugs or
> > >>   new features.
> > >>
> > >> - us...@nifi.apache.org:
> > >>- 553 subscribers (up 34 in the last 3 months):
> > >>- 748 emails sent to list (862 in previous quarter)
> > >>
> > >> - dev@nifi.apache.org:
> > >>- 383 subscribers (up 12 in the last 3 months):
> > >>- 604 emails sent to list (947 in previous quarter)
> > >>
> > >> - iss...@nifi.apache.org:
> > >>- 46 subscribers (up 4 in the last 3 months):
> > >>- 5775 emails sent to list (7457 in previous quarter)
> > >>
> > >>
> > >> ## JIRA activity:
> > >>
> > >> - 335 JIRA tickets created in the last 3 months
> > >> - 263 JIRA tickets closed/resolved in the last 3 

Re: [DRAFT][REPORT] Apache NiFi Board Report Oct 2017

[Rob Moran]

Joe, looks good with the couple changes Drew mentioned--thanks.

On Mon, Oct 9, 2017 at 10:30 AM Andrew Lim 
wrote:

> Noticed the following:
>
> -In Health Report section: "seeing a very strong pipeline to both both
> committer and PMC” (remove extra both)
> -"Apache NiFi 1.4.0 was released Oct 1 2017” (Change? Release Notes says
> Oct 2)
>
> Otherwise, looks great!
>
> -Drew
>
> > On Oct 9, 2017, at 10:23 AM, Marc  wrote:
> >
> > Joe
> >
> >  I think this looks great.
> >
> >  Thanks for putting this together,
> >
> >
> >
> > On Mon, Oct 9, 2017 at 10:12 AM, Joe Witt  wrote:
> >
> >> Team,
> >>
> >> Please see draft board report.  Am looking for feedback/edits quickly
> >> as this needs to be submitted by Wed.  I'll probably send it in
> >> tonight though as I might not have time over the next couple of days.
> >>
> >> Thanks
> >> Joe
> >>
> >> **==**==**==**==**
> >>
> >> ## Description:
> >> - Apache NiFi is an easy to use, powerful, and reliable system to
> >>   process and distribute data.
> >> - Apache NiFi MiNiFi is an edge data collection agent built to
> seamlessly
> >>   integrate with and leverage the command and control of NiFi. There are
> >>   both Java and C++ implementations.
> >> - Apache NiFi Registry is a centralized registry for key configuration
> >> items
> >>   including flow versions, assets, and extensions for Apache NiFi
> >>   and Apache MiNiFi.
> >> - Apache NiFi Nar Maven Plugin is a release artifact used for supporting
> >>   the NiFi classloader isolation model.
> >>
> >> ## Issues:
> >> - There are no issues requiring board attention at this time.
> >>
> >> ## Activity:
> >> - Released Apache NiFi 1.4.0 which is another very heavy feature,
> >> stability,
> >>   and bug fix release.  It includes improvements on the web UI coming
> >> from new
> >>   contributors which is a great sign.
> >> - The Apache NiFi downloads have been quite large and growing for some
> >> time.
> >>   The groundwork to solve this is the Apache NiFi Registry effort which
> is
> >>   coming along nicely and which the foundational elements of NiFi to
> >> integrate
> >>   with it are present.  We're probably looking at two more releases for
> >> that
> >>   to land and for which we can drop the NiFi build size down.
> >> - We continue to hover around the 100 outstanding pull requests number.
> >> What
> >>   is encouraging is we've seen an uptick in contributions both in code
> and
> >>   review feedback from new community members.
> >> - The Apache NiFi MiNiFi C++ agent has rapidly matured over the past
> >> couple of
> >>   months and is serving as a new source for contributions and
> >> contributors.
> >>
> >> ## Health report:
> >> - As stated in our past report, health of the community is strong and
> >>   indicators of strength continue trending in the right direction.
> Mailing
> >>   list and JIRA activity is strong. ASF Hipchat is serving as an on-ramp
> >> for
> >>   new users to our mailing list and JIRA systems. We continue to see new
> >>   users and contributors.
> >> - The PMC and committer ranks did not grow during the reporting period.
> >> This
> >>   is a reflection of the PMC not focusing enough on the topic though as
> >> we're
> >>   seeing a very strong pipeline to both both committer and PMC. Expect
> >> both
> >>   new committers and PMC members during our next reporting period.
> >>
> >> ## PMC changes:
> >>
> >> - Currently 22 PMC members.
> >> - Last PMC member added Fri May 26 2017.
> >>
> >> ## Committer base changes:
> >>
> >> - Currently 34 committers.
> >> - Last committer added Thu Jun 01 2017.
> >>
> >> ## Releases:
> >>
> >> - Apache NiFi 1.4.0 was released Oct 1 2017.
> >>
> >> ## Mailing list activity:
> >>
> >> - Activity on the mailing lists remains extremely high with a mixture
> >>   of new users, contributors, and deeper more experienced users and
> >>   contributors sparking discussion and questions and filing bugs or
> >>   new features.
> >>
> >> - us...@nifi.apache.org:
> >>- 553 subscribers (up 34 in the last 3 months):
> >>- 748 emails sent to list (862 in previous quarter)
> >>
> >> - dev@nifi.apache.org:
> >>- 383 subscribers (up 12 in the last 3 months):
> >>- 604 emails sent to list (947 in previous quarter)
> >>
> >> - iss...@nifi.apache.org:
> >>- 46 subscribers (up 4 in the last 3 months):
> >>- 5775 emails sent to list (7457 in previous quarter)
> >>
> >>
> >> ## JIRA activity:
> >>
> >> - 335 JIRA tickets created in the last 3 months
> >> - 263 JIRA tickets closed/resolved in the last 3 months
> >>
>
> --
Rob

Re: [DRAFT][REPORT] Apache NiFi Board Report Oct 2017

[Andrew Lim]

Noticed the following:

-In Health Report section:  "seeing a very strong pipeline to both both 
committer and PMC”(remove extra both)
-"Apache NiFi 1.4.0 was released Oct 1 2017”  (Change? Release Notes says Oct 2)

Otherwise, looks great!

-Drew

> On Oct 9, 2017, at 10:23 AM, Marc  wrote:
> 
> Joe
> 
>  I think this looks great.
> 
>  Thanks for putting this together,
> 
> 
> 
> On Mon, Oct 9, 2017 at 10:12 AM, Joe Witt  wrote:
> 
>> Team,
>> 
>> Please see draft board report.  Am looking for feedback/edits quickly
>> as this needs to be submitted by Wed.  I'll probably send it in
>> tonight though as I might not have time over the next couple of days.
>> 
>> Thanks
>> Joe
>> 
>> **==**==**==**==**
>> 
>> ## Description:
>> - Apache NiFi is an easy to use, powerful, and reliable system to
>>   process and distribute data.
>> - Apache NiFi MiNiFi is an edge data collection agent built to seamlessly
>>   integrate with and leverage the command and control of NiFi. There are
>>   both Java and C++ implementations.
>> - Apache NiFi Registry is a centralized registry for key configuration
>> items
>>   including flow versions, assets, and extensions for Apache NiFi
>>   and Apache MiNiFi.
>> - Apache NiFi Nar Maven Plugin is a release artifact used for supporting
>>   the NiFi classloader isolation model.
>> 
>> ## Issues:
>> - There are no issues requiring board attention at this time.
>> 
>> ## Activity:
>> - Released Apache NiFi 1.4.0 which is another very heavy feature,
>> stability,
>>   and bug fix release.  It includes improvements on the web UI coming
>> from new
>>   contributors which is a great sign.
>> - The Apache NiFi downloads have been quite large and growing for some
>> time.
>>   The groundwork to solve this is the Apache NiFi Registry effort which is
>>   coming along nicely and which the foundational elements of NiFi to
>> integrate
>>   with it are present.  We're probably looking at two more releases for
>> that
>>   to land and for which we can drop the NiFi build size down.
>> - We continue to hover around the 100 outstanding pull requests number.
>> What
>>   is encouraging is we've seen an uptick in contributions both in code and
>>   review feedback from new community members.
>> - The Apache NiFi MiNiFi C++ agent has rapidly matured over the past
>> couple of
>>   months and is serving as a new source for contributions and
>> contributors.
>> 
>> ## Health report:
>> - As stated in our past report, health of the community is strong and
>>   indicators of strength continue trending in the right direction. Mailing
>>   list and JIRA activity is strong. ASF Hipchat is serving as an on-ramp
>> for
>>   new users to our mailing list and JIRA systems. We continue to see new
>>   users and contributors.
>> - The PMC and committer ranks did not grow during the reporting period.
>> This
>>   is a reflection of the PMC not focusing enough on the topic though as
>> we're
>>   seeing a very strong pipeline to both both committer and PMC. Expect
>> both
>>   new committers and PMC members during our next reporting period.
>> 
>> ## PMC changes:
>> 
>> - Currently 22 PMC members.
>> - Last PMC member added Fri May 26 2017.
>> 
>> ## Committer base changes:
>> 
>> - Currently 34 committers.
>> - Last committer added Thu Jun 01 2017.
>> 
>> ## Releases:
>> 
>> - Apache NiFi 1.4.0 was released Oct 1 2017.
>> 
>> ## Mailing list activity:
>> 
>> - Activity on the mailing lists remains extremely high with a mixture
>>   of new users, contributors, and deeper more experienced users and
>>   contributors sparking discussion and questions and filing bugs or
>>   new features.
>> 
>> - us...@nifi.apache.org:
>>- 553 subscribers (up 34 in the last 3 months):
>>- 748 emails sent to list (862 in previous quarter)
>> 
>> - dev@nifi.apache.org:
>>- 383 subscribers (up 12 in the last 3 months):
>>- 604 emails sent to list (947 in previous quarter)
>> 
>> - iss...@nifi.apache.org:
>>- 46 subscribers (up 4 in the last 3 months):
>>- 5775 emails sent to list (7457 in previous quarter)
>> 
>> 
>> ## JIRA activity:
>> 
>> - 335 JIRA tickets created in the last 3 months
>> - 263 JIRA tickets closed/resolved in the last 3 months
>> 

Re: [DRAFT][REPORT] Apache NiFi Board Report Oct 2017

[Matt Burgess]

Looks good to me, thanks!

On Mon, Oct 9, 2017 at 10:12 AM, Joe Witt  wrote:
> Team,
>
> Please see draft board report.  Am looking for feedback/edits quickly
> as this needs to be submitted by Wed.  I'll probably send it in
> tonight though as I might not have time over the next couple of days.
>
> Thanks
> Joe
>
> **==**==**==**==**
>
> ## Description:
>  - Apache NiFi is an easy to use, powerful, and reliable system to
>process and distribute data.
>  - Apache NiFi MiNiFi is an edge data collection agent built to seamlessly
>integrate with and leverage the command and control of NiFi. There are
>both Java and C++ implementations.
>  - Apache NiFi Registry is a centralized registry for key configuration items
>including flow versions, assets, and extensions for Apache NiFi
>and Apache MiNiFi.
>  - Apache NiFi Nar Maven Plugin is a release artifact used for supporting
>the NiFi classloader isolation model.
>
> ## Issues:
>  - There are no issues requiring board attention at this time.
>
> ## Activity:
>  - Released Apache NiFi 1.4.0 which is another very heavy feature, stability,
>and bug fix release.  It includes improvements on the web UI coming from 
> new
>contributors which is a great sign.
>  - The Apache NiFi downloads have been quite large and growing for some time.
>The groundwork to solve this is the Apache NiFi Registry effort which is
>coming along nicely and which the foundational elements of NiFi to 
> integrate
>with it are present.  We're probably looking at two more releases for that
>to land and for which we can drop the NiFi build size down.
>  - We continue to hover around the 100 outstanding pull requests number. What
>is encouraging is we've seen an uptick in contributions both in code and
>review feedback from new community members.
>  - The Apache NiFi MiNiFi C++ agent has rapidly matured over the past couple 
> of
>months and is serving as a new source for contributions and contributors.
>
> ## Health report:
>  - As stated in our past report, health of the community is strong and
>indicators of strength continue trending in the right direction. Mailing
>list and JIRA activity is strong. ASF Hipchat is serving as an on-ramp for
>new users to our mailing list and JIRA systems. We continue to see new
>users and contributors.
>  - The PMC and committer ranks did not grow during the reporting period. This
>is a reflection of the PMC not focusing enough on the topic though as we're
>seeing a very strong pipeline to both both committer and PMC. Expect both
>new committers and PMC members during our next reporting period.
>
> ## PMC changes:
>
>  - Currently 22 PMC members.
>  - Last PMC member added Fri May 26 2017.
>
> ## Committer base changes:
>
>  - Currently 34 committers.
>  - Last committer added Thu Jun 01 2017.
>
> ## Releases:
>
>  - Apache NiFi 1.4.0 was released Oct 1 2017.
>
> ## Mailing list activity:
>
>  - Activity on the mailing lists remains extremely high with a mixture
>of new users, contributors, and deeper more experienced users and
>contributors sparking discussion and questions and filing bugs or
>new features.
>
>  - us...@nifi.apache.org:
> - 553 subscribers (up 34 in the last 3 months):
> - 748 emails sent to list (862 in previous quarter)
>
>  - dev@nifi.apache.org:
> - 383 subscribers (up 12 in the last 3 months):
> - 604 emails sent to list (947 in previous quarter)
>
>  - iss...@nifi.apache.org:
> - 46 subscribers (up 4 in the last 3 months):
> - 5775 emails sent to list (7457 in previous quarter)
>
>
> ## JIRA activity:
>
>  - 335 JIRA tickets created in the last 3 months
>  - 263 JIRA tickets closed/resolved in the last 3 months

Re: [DRAFT][REPORT] Apache NiFi Board Report Oct 2017

[Marc]

Joe

  I think this looks great.

  Thanks for putting this together,



On Mon, Oct 9, 2017 at 10:12 AM, Joe Witt  wrote:

> Team,
>
> Please see draft board report.  Am looking for feedback/edits quickly
> as this needs to be submitted by Wed.  I'll probably send it in
> tonight though as I might not have time over the next couple of days.
>
> Thanks
> Joe
>
> **==**==**==**==**
>
> ## Description:
>  - Apache NiFi is an easy to use, powerful, and reliable system to
>process and distribute data.
>  - Apache NiFi MiNiFi is an edge data collection agent built to seamlessly
>integrate with and leverage the command and control of NiFi. There are
>both Java and C++ implementations.
>  - Apache NiFi Registry is a centralized registry for key configuration
> items
>including flow versions, assets, and extensions for Apache NiFi
>and Apache MiNiFi.
>  - Apache NiFi Nar Maven Plugin is a release artifact used for supporting
>the NiFi classloader isolation model.
>
> ## Issues:
>  - There are no issues requiring board attention at this time.
>
> ## Activity:
>  - Released Apache NiFi 1.4.0 which is another very heavy feature,
> stability,
>and bug fix release.  It includes improvements on the web UI coming
> from new
>contributors which is a great sign.
>  - The Apache NiFi downloads have been quite large and growing for some
> time.
>The groundwork to solve this is the Apache NiFi Registry effort which is
>coming along nicely and which the foundational elements of NiFi to
> integrate
>with it are present.  We're probably looking at two more releases for
> that
>to land and for which we can drop the NiFi build size down.
>  - We continue to hover around the 100 outstanding pull requests number.
> What
>is encouraging is we've seen an uptick in contributions both in code and
>review feedback from new community members.
>  - The Apache NiFi MiNiFi C++ agent has rapidly matured over the past
> couple of
>months and is serving as a new source for contributions and
> contributors.
>
> ## Health report:
>  - As stated in our past report, health of the community is strong and
>indicators of strength continue trending in the right direction. Mailing
>list and JIRA activity is strong. ASF Hipchat is serving as an on-ramp
> for
>new users to our mailing list and JIRA systems. We continue to see new
>users and contributors.
>  - The PMC and committer ranks did not grow during the reporting period.
> This
>is a reflection of the PMC not focusing enough on the topic though as
> we're
>seeing a very strong pipeline to both both committer and PMC. Expect
> both
>new committers and PMC members during our next reporting period.
>
> ## PMC changes:
>
>  - Currently 22 PMC members.
>  - Last PMC member added Fri May 26 2017.
>
> ## Committer base changes:
>
>  - Currently 34 committers.
>  - Last committer added Thu Jun 01 2017.
>
> ## Releases:
>
>  - Apache NiFi 1.4.0 was released Oct 1 2017.
>
> ## Mailing list activity:
>
>  - Activity on the mailing lists remains extremely high with a mixture
>of new users, contributors, and deeper more experienced users and
>contributors sparking discussion and questions and filing bugs or
>new features.
>
>  - us...@nifi.apache.org:
> - 553 subscribers (up 34 in the last 3 months):
> - 748 emails sent to list (862 in previous quarter)
>
>  - dev@nifi.apache.org:
> - 383 subscribers (up 12 in the last 3 months):
> - 604 emails sent to list (947 in previous quarter)
>
>  - iss...@nifi.apache.org:
> - 46 subscribers (up 4 in the last 3 months):
> - 5775 emails sent to list (7457 in previous quarter)
>
>
> ## JIRA activity:
>
>  - 335 JIRA tickets created in the last 3 months
>  - 263 JIRA tickets closed/resolved in the last 3 months
>

Re: [DRAFT][REPORT] Apache NiFi Board Report Oct 2017

[Pierre Villard]

Hey Joe,

It looks good to me.

Thanks
Pierre


2017-10-09 16:12 GMT+02:00 Joe Witt :

> Team,
>
> Please see draft board report.  Am looking for feedback/edits quickly
> as this needs to be submitted by Wed.  I'll probably send it in
> tonight though as I might not have time over the next couple of days.
>
> Thanks
> Joe
>
> **==**==**==**==**
>
> ## Description:
>  - Apache NiFi is an easy to use, powerful, and reliable system to
>process and distribute data.
>  - Apache NiFi MiNiFi is an edge data collection agent built to seamlessly
>integrate with and leverage the command and control of NiFi. There are
>both Java and C++ implementations.
>  - Apache NiFi Registry is a centralized registry for key configuration
> items
>including flow versions, assets, and extensions for Apache NiFi
>and Apache MiNiFi.
>  - Apache NiFi Nar Maven Plugin is a release artifact used for supporting
>the NiFi classloader isolation model.
>
> ## Issues:
>  - There are no issues requiring board attention at this time.
>
> ## Activity:
>  - Released Apache NiFi 1.4.0 which is another very heavy feature,
> stability,
>and bug fix release.  It includes improvements on the web UI coming
> from new
>contributors which is a great sign.
>  - The Apache NiFi downloads have been quite large and growing for some
> time.
>The groundwork to solve this is the Apache NiFi Registry effort which is
>coming along nicely and which the foundational elements of NiFi to
> integrate
>with it are present.  We're probably looking at two more releases for
> that
>to land and for which we can drop the NiFi build size down.
>  - We continue to hover around the 100 outstanding pull requests number.
> What
>is encouraging is we've seen an uptick in contributions both in code and
>review feedback from new community members.
>  - The Apache NiFi MiNiFi C++ agent has rapidly matured over the past
> couple of
>months and is serving as a new source for contributions and
> contributors.
>
> ## Health report:
>  - As stated in our past report, health of the community is strong and
>indicators of strength continue trending in the right direction. Mailing
>list and JIRA activity is strong. ASF Hipchat is serving as an on-ramp
> for
>new users to our mailing list and JIRA systems. We continue to see new
>users and contributors.
>  - The PMC and committer ranks did not grow during the reporting period.
> This
>is a reflection of the PMC not focusing enough on the topic though as
> we're
>seeing a very strong pipeline to both both committer and PMC. Expect
> both
>new committers and PMC members during our next reporting period.
>
> ## PMC changes:
>
>  - Currently 22 PMC members.
>  - Last PMC member added Fri May 26 2017.
>
> ## Committer base changes:
>
>  - Currently 34 committers.
>  - Last committer added Thu Jun 01 2017.
>
> ## Releases:
>
>  - Apache NiFi 1.4.0 was released Oct 1 2017.
>
> ## Mailing list activity:
>
>  - Activity on the mailing lists remains extremely high with a mixture
>of new users, contributors, and deeper more experienced users and
>contributors sparking discussion and questions and filing bugs or
>new features.
>
>  - us...@nifi.apache.org:
> - 553 subscribers (up 34 in the last 3 months):
> - 748 emails sent to list (862 in previous quarter)
>
>  - dev@nifi.apache.org:
> - 383 subscribers (up 12 in the last 3 months):
> - 604 emails sent to list (947 in previous quarter)
>
>  - iss...@nifi.apache.org:
> - 46 subscribers (up 4 in the last 3 months):
> - 5775 emails sent to list (7457 in previous quarter)
>
>
> ## JIRA activity:
>
>  - 335 JIRA tickets created in the last 3 months
>  - 263 JIRA tickets closed/resolved in the last 3 months
>

[DRAFT][REPORT] Apache NiFi Board Report Oct 2017

[Joe Witt]

Team,

Please see draft board report.  Am looking for feedback/edits quickly
as this needs to be submitted by Wed.  I'll probably send it in
tonight though as I might not have time over the next couple of days.

Thanks
Joe

**==**==**==**==**

## Description:
 - Apache NiFi is an easy to use, powerful, and reliable system to
   process and distribute data.
 - Apache NiFi MiNiFi is an edge data collection agent built to seamlessly
   integrate with and leverage the command and control of NiFi. There are
   both Java and C++ implementations.
 - Apache NiFi Registry is a centralized registry for key configuration items
   including flow versions, assets, and extensions for Apache NiFi
   and Apache MiNiFi.
 - Apache NiFi Nar Maven Plugin is a release artifact used for supporting
   the NiFi classloader isolation model.

## Issues:
 - There are no issues requiring board attention at this time.

## Activity:
 - Released Apache NiFi 1.4.0 which is another very heavy feature, stability,
   and bug fix release.  It includes improvements on the web UI coming from new
   contributors which is a great sign.
 - The Apache NiFi downloads have been quite large and growing for some time.
   The groundwork to solve this is the Apache NiFi Registry effort which is
   coming along nicely and which the foundational elements of NiFi to integrate
   with it are present.  We're probably looking at two more releases for that
   to land and for which we can drop the NiFi build size down.
 - We continue to hover around the 100 outstanding pull requests number. What
   is encouraging is we've seen an uptick in contributions both in code and
   review feedback from new community members.
 - The Apache NiFi MiNiFi C++ agent has rapidly matured over the past couple of
   months and is serving as a new source for contributions and contributors.

## Health report:
 - As stated in our past report, health of the community is strong and
   indicators of strength continue trending in the right direction. Mailing
   list and JIRA activity is strong. ASF Hipchat is serving as an on-ramp for
   new users to our mailing list and JIRA systems. We continue to see new
   users and contributors.
 - The PMC and committer ranks did not grow during the reporting period. This
   is a reflection of the PMC not focusing enough on the topic though as we're
   seeing a very strong pipeline to both both committer and PMC. Expect both
   new committers and PMC members during our next reporting period.

## PMC changes:

 - Currently 22 PMC members.
 - Last PMC member added Fri May 26 2017.

## Committer base changes:

 - Currently 34 committers.
 - Last committer added Thu Jun 01 2017.

## Releases:

 - Apache NiFi 1.4.0 was released Oct 1 2017.

## Mailing list activity:

 - Activity on the mailing lists remains extremely high with a mixture
   of new users, contributors, and deeper more experienced users and
   contributors sparking discussion and questions and filing bugs or
   new features.

 - us...@nifi.apache.org:
- 553 subscribers (up 34 in the last 3 months):
- 748 emails sent to list (862 in previous quarter)

 - dev@nifi.apache.org:
- 383 subscribers (up 12 in the last 3 months):
- 604 emails sent to list (947 in previous quarter)

 - iss...@nifi.apache.org:
- 46 subscribers (up 4 in the last 3 months):
- 5775 emails sent to list (7457 in previous quarter)


## JIRA activity:

 - 335 JIRA tickets created in the last 3 months
 - 263 JIRA tickets closed/resolved in the last 3 months

Re: route flow based on variable

[尹文才]

Thanks Bryan, I tried after reading your reply, I could use the variable
directly in RouteOnAttribute, I thought I could only use attributes of a
FlowFile in RouteOnAttribute, thanks.
/Ben

2017-10-09 19:24 GMT+08:00 Bryan Bende :

> Ben,
>
> 1) Yes, the variables are hierarchical, so a variable at the root group
> would be visible to all components, unless there is a variable with the
> same name at a lower level which would override it.
>
> 2) I haven’t tried this, but I would expect that you should still be able
> to use RouteOnAttribute to route on a variable… Lets say that your root
> group has a variable “env” and in one environment you have this set to
> “dev” and in another environment you have it set to “prod”. You might have
> a part of your flow that you only run in “prod” so you put a
> RouteOnAttribute with something like ${env:equals(“prod”)} which would only
> enable this path in prod.
>
> Variables on their own are not associated with flow files, so if you
> wanted to do something more specific per-flow file, then the only way would
> be what you described with using UpdateAttribute.
>
> Thanks,
>
> Bryan
>
> > On Oct 8, 2017, at 10:44 PM, 尹文才  wrote:
> >
> > Hi guys, I've played around with the latest NIFI 1.4.0 release for a
> while
> > and I think the new variable registry feature is great, however I have 2
> > questions about this feature:
> >
> > 1. It seems that I could only add variables to a processor group, could I
> > add a global variable in the NIFI root processor group so it could be
> used
> > anywhere inside NIFI?
> >
> > 2. I want to route to different flows based on the variables I added, but
> > currently the only way I know that could make this work is like this:
> > myprocessor->updateAttribute(add the variable into FlowFile attribute)->
> > routeOnAttribute->different flows based on the variable value
> >
> > I didn't find any routeOnVariable processor, is there any easier way
> that I
> > could use to implement conditional flow in NIFI? Thanks
> >
> > /Ben
>
>

Re: Long recovery time

[Joe Gresock]

Thanks for the input, Mark.  I also enabled DEBUG logging on
MinimalLockingWriteAheadLog
and that did show me that it was making progress in recovery.  The node did
come up overnight and is back down to a reasonable flowfile_repository size.

On Mon, Oct 9, 2017 at 12:33 AM, Mark Payne  wrote:

> Joe,
>
> When you have a huge FlowFile repository like that, it can indeed take
> quite a long time to recover (potentially a few hours).
> But I say that with the caveat that the FlowFile repository should
> probably never reach that size.
> It generally doesn't grow beyond a couple of GB. The two things that I
> have seen cause tremendous
> growth in the FlowFile repository are OutOfMemoryError and Too Many Open
> File IOExceptions. Either could prevent
> the FlowFile Repository from properly checkpointing, which would cause it
> to grow unbounded. The
> former case (OOME), though, is far more likely. The reason that I say this
> is that if any OOME (or any uncaught Throwable)
> is thrown when checkpointing the FlowFile repository, that background
> thread will die. I did submit a fix for that,
> and it was included in 1.4.0, I believe (ASF JIRA is having maintenance
> performed at the moment I believe, so I'm
> not able to look it up at the moment).
>
> There probably are some things that can be done to address the super long
> recovery time, though. The problem should
> be easy to replicate by telling NiFi to checkpoint the flowfile repository
> once an hour instead of the default once every 2
> minutes (via the 'nifi.flowfile.repository.checkpoint.interval' property)
> so that would allow us to know what is taking so
> long and address accordingly.
>
>
> Thanks
> -Mark
>
>
> > On Oct 8, 2017, at 11:53 AM, Joe Gresock  wrote:
> >
> > I have a a NiFi 1.1.0 instance whose disk nearly (but not quite) filled
> > up.  I noticed that some of its NiFi processors were hanging so I
> restarted
> > it, but it's taking over an hour to come back up.
> >
> > My question is: how can I tell if NiFi is doing something productive (and
> > therefore I should just let it finish) vs. hanging (and therefore I
> should
> > try something else)?  Is it possible that NiFi could take hours to stand
> > back up?  My content_repository is 276GB and my flowfile_repository is
> > 640GB.
> >
> > I see the following in the logs:
> >
> > o.a.n.controller.StandardFlowFileQueue Recovered 8 swap files for
> > FlowFileQueue[...] in 51 millis
> > org.wali.MinimalLockingWriteAheadLog finished recovering records.
> > Performing Checkpoint to ensure proper state of Partitions before updates
> > org.wali.MinimalLockingWriteAheadLog Successfully recovered 10536141
> > records in 38509 milliseconds
> >
> > Thereafter, the only thing I see in the logs are these periodic messages:
> > org.wali.MinimalLockingWriteAheadLog checkpointed with 2 Records and 0
> Swap
> > Files in 16 milliseconds, Max Transaction ID 31
> >
> > I did a thread dump and see pretty standard stuff, including one that I
> > thought might be relevant:
> > "main" Id=1 RUNNABLE
> >  at java.util.HashMap.putVal(HashMap.java:641)
> >  at java.util.HashMap.put(HashMap.java:611)
> >  at
> > org.apache.nifi.repository.schema.SchemaRecordReader.readFieldValue(
> SchemaRecordReader.java:154)
> >
> > I took a couple dumps in a row in case it was hung here, but it appears
> to
> > be progressing to different points in the stack.
> >
> > NiFi is the only thing running on this instance, and nearly all of its
> 48GB
> > of RAM are being used, and I did notice that it is doing some heavy reads
> > but not many writes (using iostat).
> >
> > Thanks,
> > Joe
> > --
> > I know what it is to be in need, and I know what it is to have plenty.  I
> > have learned the secret of being content in any and every situation,
> > whether well fed or hungry, whether living in plenty or in want.  I can
> do
> > all this through him who gives me strength.*-Philippians 4:12-13*
>
>


-- 
I know what it is to be in need, and I know what it is to have plenty.  I
have learned the secret of being content in any and every situation,
whether well fed or hungry, whether living in plenty or in want.  I can do
all this through him who gives me strength.*-Philippians 4:12-13*

Re: FuzzyHashContent/CompareFuzzyHash processor

[shankhamajumdar]

Hi Andy,

Thanks for the reply. But I am still not able to solve my use case. For
example 

I have a data file in the below structure.

Col1  Col2  Col3  Col4  Col5

Test1Test2 Test3 Test4 Test5

I want to do a fuzzy matching on Col2 and Col3 and generate an output file.

I am using getFile and FuzzyHashContent processor but not able to design the
flow. Need your help on this.

Regards,
Shankha






--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

Can we archive multiple user search base in login-identity-providers.xml NIFI

[pramod s]

Hello,

I have setup three node secure NIFI cluster and integrated with
LDAP for user login .

my doubts
*-*-*-*-*-*--*
1)In login-identity-providers.xml can we add multiple USER Search Base of
ldap.(i tried by adding multiple usersearchbase but failed)

as I attached screen shot in that can we add multiple USer search BAse.

because LDAP user search should happen on multiple sub domains eg :
DC=example1,dc=example,dc=com DC=example2,dc=example,dc=com

3) TO achieve mutitenancy in ldap what is the configuration changes we need
to make

My refernce URL:
-**-*-*-*-*-*-*-*-*-*-
1.
https://pierrevillard.com/2016/11/29/apache-nifi-1-1-0-secured-cluster-setup/
2.
https://bryanbende.com/development/2016/08/17/apache-nifi-1-0-0-authorization-and-multi-tenancy
3.https://pierrevillard.com/2017/01/24/integration-of-nifi-with-ldap/

Re: Funnel Queue Slowness

[Bryan Bende]

Peter,

The images didn’t come across for me, but since you mentioned that a failure 
queue is involved, is it possible all the flow files going to failure are being 
penalized which would cause them to not be processed immediately?

-Bryan

> On Oct 8, 2017, at 10:49 PM, Peter Wicks (pwicks)  wrote:
> 
> I’ve been running into an issue on 1.4.0 where my Funnel sometimes runs slow. 
> I haven’t been able to create a nice reproducible test case to pass on.
> What I’m seeing is that my failure queue on the right will start to fill up, 
> even though there is plenty of room for them in the next queue. You can see 
> that the Tasks/Time is fairly low, only 24 in the last 5 minutes (first 
> image), so it’s not that the FlowFile’s are moving so fast that they just 
> appear to be in queue.
> 
> If I stop the downstream processor the files slowly trickle through the 
> funnel into the next queue slowly. I had an Oldest FlowFile First prioritizer 
> on the downstream queue. I tried removing it but there was no change in 
> behavior.
> One time where I saw this behavior in the past was when my NiFi instance was 
> thread starved, but there are plenty of threads available on the instance and 
> all other processors are running fine. I also don’t understand why it 
> trickles the FlowFile’s in, from what I’ve seen in the code Funnel grabs 
> large batches at one time…
> 
> Thoughts?
> 
> (Sometimes my images don’t make it, let me know if that happens.)
> 



signature.asc
Description: Message signed with OpenPGP

Re: ListenTcpRecord

[Bryan Bende]

Clay,

Multiple packets should not be an issue since it is reading a stream of data 
from the socket, but I don’t think the prefixed length will work.

The data coming across has to be in a format that one of the record readers can 
understand. If you have JSON data and then have something additional added to 
it like the length, then its not going to be readable by a standard JSON 
library.

You could potentially create a custom record reader to handle your format.

-Bryan

> On Oct 7, 2017, at 10:37 AM, Clay Teahouse  wrote:
> 
> Thank you, Bryan. I was able to set up the flow using your template and
> then fix my own.
> 
> Can ListenTCPRecord deal with records spread across multiple packets? More
> specifically, can it handle payloads prefixed with the length? I receive
> JSON and protobuf messages from a client and each record is prefixed with
> the length of the message.
> 
> -Clay
> 
> On Thu, Oct 5, 2017 at 8:22 AM, Bryan Bende  wrote:
> 
>> Have you tried using the template?
>> 
>> https://gist.githubusercontent.com/bbende/fa2bff34e721fef21453986336664c
>> b2/raw/db658c64f75fec47785ab63920ee23582bf1492f/multi_line_log_
>> processing.xml
>> 
>> If you import that it will give you the exact flow from my post and
>> all you have to do is start everything.
>> 
>> PutTCP is sending to ListenTCP, the connection coming out of PutTCP to
>> LogAttribute is for the failure relationship just to easily see if
>> anything fails.
>> 
>> -Bryan
>> 
>> 
>> On Thu, Oct 5, 2017 at 7:05 AM, Clay Teahouse 
>> wrote:
>>> Thanks Bryan, for the feedback.
>>> 
>>> I don't seem to be able  to replicate the example in your blog. So, is
>> the
>>> flow  GenerateFlowFile -> TCPPUT --> LogAttribute and ListenTCPRecord
>>> -->LogAttribute?  Both TCPPut and ListenTCPRecord  are sending data to
>>> LogAttribute?  Shouldn't ListenTCPRecord be receiving the data data?
>> Also,
>>> I am not able to link ListenTCPRecord to any other processor.
>>> 
>>> In any case, I tried to send logs to ListenTCPRecord using netcat and
>> have
>>> ListenTCPRecord send the processed data to LogAttribute, but
>>> ListenTCPRecord doesn't seem to be listening (I don't see any activity on
>>> the processor). I'd appreciate if you let me what I am doing wrong.
>>> 
>>> thanks
>>> Clay
>>> 
>>> On Wed, Oct 4, 2017 at 9:14 AM, Bryan Bende  wrote:
>>> 
 Hello,
 
 I wrote a post that shows an example of using ListenTCPRecord with a
 GrokReader to receive multi-line log messages. There is a link to a
 template of the flow at the very end.
 
 You could easily change the example so that PutTCP is sending a single
 JSON document, or an array of JSON documents, and ListenTCPRecord is
 using a JsonTreeReader.
 
 We don't have a protobuf record reader so that currently isn't an
>> option.
 
 Let us know if you have any other questions.
 
 -Bryan
 
 [1] https://bryanbende.com/development/2017/10/04/apache-
 nifi-processing-multiline-logs
 
 On Wed, Oct 4, 2017 at 8:39 AM, Clay Teahouse 
 wrote:
> Hi All,
> 
> Does anyone have an example of ListenTcpRecord processor in action,
>> say
 for
> example, with a json or protobuf reader? I am specifically wondering
 about
> the record length/prefix.
> 
> thanks
> Clay
 
>> 



signature.asc
Description: Message signed with OpenPGP

Re: route flow based on variable

[Bryan Bende]

Ben,

1) Yes, the variables are hierarchical, so a variable at the root group would 
be visible to all components, unless there is a variable with the same name at 
a lower level which would override it.

2) I haven’t tried this, but I would expect that you should still be able to 
use RouteOnAttribute to route on a variable… Lets say that your root group has 
a variable “env” and in one environment you have this set to “dev” and in 
another environment you have it set to “prod”. You might have a part of your 
flow that you only run in “prod” so you put a RouteOnAttribute with something 
like ${env:equals(“prod”)} which would only enable this path in prod.

Variables on their own are not associated with flow files, so if you wanted to 
do something more specific per-flow file, then the only way would be what you 
described with using UpdateAttribute.

Thanks,

Bryan

> On Oct 8, 2017, at 10:44 PM, 尹文才  wrote:
> 
> Hi guys, I've played around with the latest NIFI 1.4.0 release for a while
> and I think the new variable registry feature is great, however I have 2
> questions about this feature:
> 
> 1. It seems that I could only add variables to a processor group, could I
> add a global variable in the NIFI root processor group so it could be used
> anywhere inside NIFI?
> 
> 2. I want to route to different flows based on the variables I added, but
> currently the only way I know that could make this work is like this:
> myprocessor->updateAttribute(add the variable into FlowFile attribute)->
> routeOnAttribute->different flows based on the variable value
> 
> I didn't find any routeOnVariable processor, is there any easier way that I
> could use to implement conditional flow in NIFI? Thanks
> 
> /Ben



signature.asc
Description: Message signed with OpenPGP