[GitHub] nifi pull request: NIFI-1520 by default skip javadoc/source jar ge...

[trkurc]

Github user trkurc commented on the pull request:

https://github.com/apache/nifi/pull/234#issuecomment-185541846
  
applies clean to support/nifi-0.5.x and master. running through a test 
release:prepare and release:perform to verify correct behavior with those 
plugins


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1520 by default skip javadoc/source jar ge...

[trkurc]

Github user trkurc commented on the pull request:

https://github.com/apache/nifi/pull/234#issuecomment-185538294
  
reviewing now


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1520 by default skip javadoc/source jar ge...

[joewitt]

GitHub user joewitt opened a pull request:

https://github.com/apache/nifi/pull/234

NIFI-1520 by default skip javadoc/source jar generation in nars/wars



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/joewitt/incubator-nifi NIFI-1520

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/234.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #234


commit b51aff6d4cf2b1ef9e4c9b592b398e09ff8a24cb
Author: joewitt 
Date:   2016-02-18T04:25:11Z

NIFI-1520 by default skip javadoc and source jar generation in nars and wars




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Git NiFi processor

[xmlking]

I have an usecase where I have to pull some configuration files from Git 
repository and store them into a server directory, whenever files are updated 
via webhooks.[automated deployment]

Checking if anybody build custom processor to pull files from Git.

By the way, it would be nice if NiFi UI directly provide saving and versioning 
flow templates into Git.
Thanks 
Sumo

[GitHub] nifi pull request: NIFI-614 Added initial support for new style JM...

[trkurc]

Github user trkurc commented on a diff in the pull request:

https://github.com/apache/nifi/pull/222#discussion_r53268977
  
--- Diff: nifi-nar-bundles/nifi-jms-bundle/nifi-jms-cf-service/.gitignore 
---
@@ -0,0 +1 @@
+/lib/
--- End diff --

why is this necessary?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] NiFi 0.5.1 release

[Joe Witt]

+1

On Wed, Feb 17, 2016 at 10:12 PM, Tony Kurc  wrote:
> All,
> Just so we're clear, my expectations is that the fixes we need above will
> go into master (0.6.0-SNAPSHOT). I (or the contributor) will attempt to
> also apply those fixes to a branch I made off of nifi-0.5.0, called
> "support/nifi-0.5.x", and when done we can test and when satisfied, start
> the 0.5.1 release process.
>
>
> On Wed, Feb 17, 2016 at 8:28 PM, Tony Kurc  wrote:
>
>> I lieu of a decision on a git branching strategy, does anyone object to me
>> naming a branch "support/0.5.x" so I can start pulling together 0.5.1?
>>
>> On Wed, Feb 17, 2016 at 8:17 PM, Joe Witt  wrote:
>>
>>> Team,
>>>
>>> First, congrats on getting 0.5.0 out the door.  Very nice release with
>>> excellent new features, enhancements, and bug fixes with involvement
>>> of many new contributors.
>>>
>>> The effort toward 0.6.0 is well underway and if we are shooting for
>>> our original target would still arrive around mid march-ish.  Seems
>>> reasonable given progress so far.
>>>
>>> That said, I'd like to propose we go ahead and put out an 0.5.1 as
>>> soon as the patches are addressed.  The finding that Lars helped
>>> expose and that Mark Payne is now working  [1] in my view warrants a
>>> release by itself as it is data loss related.  Also, we can address
>>> the work Matt Gilman is doing to ensure properly authorized users can
>>> access resources they're being blocked by [2] and the license/notice
>>> findings Sean discovered during the last release [3].  There are a
>>> couple odd/ends in the list as well [4].  Purely a bug fix release as
>>> spec'd out in release notes [5].
>>>
>>> I checked with Tony and he is happy to continue performing RM tasks on
>>> the 0.5.x line.
>>>
>>> [1] https://issues.apache.org/jira/browse/NIFI-1527
>>> [2] https://issues.apache.org/jira/browse/NIFI-1497
>>> [3] https://issues.apache.org/jira/browse/NIFI-1520
>>> [4] https://s.apache.org/nifi-0.5.1-candidate
>>> [5] https://cwiki.apache.org/confluence/display/NIFI/Release+Notes
>>>
>>> Thanks
>>> Joe
>>>
>>
>>

Re: [DISCUSS] NiFi 0.5.1 release

[Tony Kurc]

All,
Just so we're clear, my expectations is that the fixes we need above will
go into master (0.6.0-SNAPSHOT). I (or the contributor) will attempt to
also apply those fixes to a branch I made off of nifi-0.5.0, called
"support/nifi-0.5.x", and when done we can test and when satisfied, start
the 0.5.1 release process.


On Wed, Feb 17, 2016 at 8:28 PM, Tony Kurc  wrote:

> I lieu of a decision on a git branching strategy, does anyone object to me
> naming a branch "support/0.5.x" so I can start pulling together 0.5.1?
>
> On Wed, Feb 17, 2016 at 8:17 PM, Joe Witt  wrote:
>
>> Team,
>>
>> First, congrats on getting 0.5.0 out the door.  Very nice release with
>> excellent new features, enhancements, and bug fixes with involvement
>> of many new contributors.
>>
>> The effort toward 0.6.0 is well underway and if we are shooting for
>> our original target would still arrive around mid march-ish.  Seems
>> reasonable given progress so far.
>>
>> That said, I'd like to propose we go ahead and put out an 0.5.1 as
>> soon as the patches are addressed.  The finding that Lars helped
>> expose and that Mark Payne is now working  [1] in my view warrants a
>> release by itself as it is data loss related.  Also, we can address
>> the work Matt Gilman is doing to ensure properly authorized users can
>> access resources they're being blocked by [2] and the license/notice
>> findings Sean discovered during the last release [3].  There are a
>> couple odd/ends in the list as well [4].  Purely a bug fix release as
>> spec'd out in release notes [5].
>>
>> I checked with Tony and he is happy to continue performing RM tasks on
>> the 0.5.x line.
>>
>> [1] https://issues.apache.org/jira/browse/NIFI-1527
>> [2] https://issues.apache.org/jira/browse/NIFI-1497
>> [3] https://issues.apache.org/jira/browse/NIFI-1520
>> [4] https://s.apache.org/nifi-0.5.1-candidate
>> [5] https://cwiki.apache.org/confluence/display/NIFI/Release+Notes
>>
>> Thanks
>> Joe
>>
>
>

[GitHub] nifi pull request: Fix NiFi-1501 : Test Monitor Activity has spuri...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/225


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: Fix NiFi-1501 : Test Monitor Activity has spuri...

[trkurc]

Github user trkurc commented on the pull request:

https://github.com/apache/nifi/pull/225#issuecomment-185514088
  
reviewing now.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1491

[trkurc]

Github user trkurc commented on the pull request:

https://github.com/apache/nifi/pull/227#issuecomment-185507765
  
reviewing now


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1063

[trkurc]

Github user trkurc commented on the pull request:

https://github.com/apache/nifi/pull/226#issuecomment-185507421
  
Merged in! Thanks for the contribution!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1063

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/226


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] NiFi 0.5.1 release

[Tony Kurc]

I lieu of a decision on a git branching strategy, does anyone object to me
naming a branch "support/0.5.x" so I can start pulling together 0.5.1?

On Wed, Feb 17, 2016 at 8:17 PM, Joe Witt  wrote:

> Team,
>
> First, congrats on getting 0.5.0 out the door.  Very nice release with
> excellent new features, enhancements, and bug fixes with involvement
> of many new contributors.
>
> The effort toward 0.6.0 is well underway and if we are shooting for
> our original target would still arrive around mid march-ish.  Seems
> reasonable given progress so far.
>
> That said, I'd like to propose we go ahead and put out an 0.5.1 as
> soon as the patches are addressed.  The finding that Lars helped
> expose and that Mark Payne is now working  [1] in my view warrants a
> release by itself as it is data loss related.  Also, we can address
> the work Matt Gilman is doing to ensure properly authorized users can
> access resources they're being blocked by [2] and the license/notice
> findings Sean discovered during the last release [3].  There are a
> couple odd/ends in the list as well [4].  Purely a bug fix release as
> spec'd out in release notes [5].
>
> I checked with Tony and he is happy to continue performing RM tasks on
> the 0.5.x line.
>
> [1] https://issues.apache.org/jira/browse/NIFI-1527
> [2] https://issues.apache.org/jira/browse/NIFI-1497
> [3] https://issues.apache.org/jira/browse/NIFI-1520
> [4] https://s.apache.org/nifi-0.5.1-candidate
> [5] https://cwiki.apache.org/confluence/display/NIFI/Release+Notes
>
> Thanks
> Joe
>

Re: [DISCUSS] NiFi 0.5.1 release

[Matt Burgess]

Sounds good to me (non-binding ;)

Sent from my iPhone

> On Feb 17, 2016, at 8:17 PM, Joe Witt  wrote:
> 
> Team,
> 
> First, congrats on getting 0.5.0 out the door.  Very nice release with
> excellent new features, enhancements, and bug fixes with involvement
> of many new contributors.
> 
> The effort toward 0.6.0 is well underway and if we are shooting for
> our original target would still arrive around mid march-ish.  Seems
> reasonable given progress so far.
> 
> That said, I'd like to propose we go ahead and put out an 0.5.1 as
> soon as the patches are addressed.  The finding that Lars helped
> expose and that Mark Payne is now working  [1] in my view warrants a
> release by itself as it is data loss related.  Also, we can address
> the work Matt Gilman is doing to ensure properly authorized users can
> access resources they're being blocked by [2] and the license/notice
> findings Sean discovered during the last release [3].  There are a
> couple odd/ends in the list as well [4].  Purely a bug fix release as
> spec'd out in release notes [5].
> 
> I checked with Tony and he is happy to continue performing RM tasks on
> the 0.5.x line.
> 
> [1] https://issues.apache.org/jira/browse/NIFI-1527
> [2] https://issues.apache.org/jira/browse/NIFI-1497
> [3] https://issues.apache.org/jira/browse/NIFI-1520
> [4] https://s.apache.org/nifi-0.5.1-candidate
> [5] https://cwiki.apache.org/confluence/display/NIFI/Release+Notes
> 
> Thanks
> Joe

[DISCUSS] NiFi 0.5.1 release

[Joe Witt]

Team,

First, congrats on getting 0.5.0 out the door.  Very nice release with
excellent new features, enhancements, and bug fixes with involvement
of many new contributors.

The effort toward 0.6.0 is well underway and if we are shooting for
our original target would still arrive around mid march-ish.  Seems
reasonable given progress so far.

That said, I'd like to propose we go ahead and put out an 0.5.1 as
soon as the patches are addressed.  The finding that Lars helped
expose and that Mark Payne is now working  [1] in my view warrants a
release by itself as it is data loss related.  Also, we can address
the work Matt Gilman is doing to ensure properly authorized users can
access resources they're being blocked by [2] and the license/notice
findings Sean discovered during the last release [3].  There are a
couple odd/ends in the list as well [4].  Purely a bug fix release as
spec'd out in release notes [5].

I checked with Tony and he is happy to continue performing RM tasks on
the 0.5.x line.

[1] https://issues.apache.org/jira/browse/NIFI-1527
[2] https://issues.apache.org/jira/browse/NIFI-1497
[3] https://issues.apache.org/jira/browse/NIFI-1520
[4] https://s.apache.org/nifi-0.5.1-candidate
[5] https://cwiki.apache.org/confluence/display/NIFI/Release+Notes

Thanks
Joe

[ANNOUNCE] Apache NiFi 0.5.0 release

[Tony Kurc]

Hello,

The Apache NiFi team would like to announce the release of Apache NiFi
0.5.0.

Apache NiFi is an easy to use, powerful, and reliable system to process and
distribute data.  Apache NiFi was made for dataflow.  It supports highly
configurable directed graphs of data routing, transformation, and system
mediation logic.

More details on Apache NiFi can be found here:
http://nifi.apache.org/

The release artifacts can be downloaded from here:
http://nifi.apache.org/download.html

Maven artifacts have been made available here:
https://repository.apache.org/content/repositories/releases/org/apache/nifi/

106 issues were closed/resolved for this release:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12334158

Release note highlights can be found here:
https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version0.5.0

Migration guidance here:
https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance

Thank you
The Apache NiFi team

Re: FlowFiles missing (MissingFlowFileException)

[Lars Francke]

Mark, Joe,

thanks a lot for investigating, reproducing and explaining what's going on,
I couldn't have done that.

I built the 0.5.0 RC3 version from the git commit mentioned and haven't
changed a single configuration property. I hadn't even heard of archiving
so far. Just looked at the properties to be safe and archiving is enabled
and threshold is set to 50%. My local disk is definitely more than 50%
utilised.

Had the archive worked would it have been a manual process to recover the
data?

It looks like you've got it covered but if there's anything else I can do
to help let me know.

Lars

On Wed, Feb 17, 2016 at 8:16 PM, Mark Payne  wrote:

> Lars,
>
> Joe is correct - i felt a need for coffee before trying to identify the
> exact case
> that will cause this issue to occur :)
>
> I created a ticket for this issue here:
> https://issues.apache.org/jira/browse/NIFI-1527 <
> https://issues.apache.org/jira/browse/NIFI-1527>
>
> I share your opinion that the funnel is not related. Rather, I believe the
> issue has to do
> with swapping of data. NiFi has a mechanism for swapping out FlowFiles to
> disk, rather
> than leaving them all in memory (here we are talking about the FlowFile
> objects, which
> contains things of attributes. The content is already written to disk and
> not kept in memory).
>
> With the default settings, this happens if at least 30,000 FlowFiles exist
> in the same queue.
> In this case, upon restart, if all of the FlowFiles that reference a
> specific "content file" on disk
> are swapped out, the Content Repository will end up removing or archiving
> that data. If the
> archive is disabled or full, the repository will end up removing it. I
> believe this is what you saw.
> I believe the content was removed on restart because all FlowFiles that
> referenced it were
> swapped out.
>
> Please confirm what Joe asked below: that you either disabled archiving or
> that your disk is
> at least 50% full (or that you changed that configuration parameter in
> conf/nifi.properties).
>
> We will certainly be addressing this issue very promptly. A huge thank you
> for noticing the weirdness
> and bringing it to our attention and for providing such great details!
>
> -Mark
>
>
> > On Feb 17, 2016, at 1:48 PM, Joe Witt  wrote:
> >
> > Lars,
> >
> > First of all thank you very much for reporting this and providing the
> > detail you did.
> >
> > Mark Payne just replicated the problem it sounds like then rather than
> > emailing he decided to go get coffee :-).
> >
> > We will be working this up for very prompt resolution and it warrants
> > a release in my view.
> >
> > Can you confirm that you are either not using archiving or you are
> > using it and you have more than 50% of space on the partition nifi is
> > running used up?
> >
> > I'll let Mark share the details.
> >
> > Thanks
> > Joe
> >
> > On Wed, Feb 17, 2016 at 4:06 AM, Lars Francke 
> wrote:
> >> It seems as if the Funnel thing wasn't actually the problem.
> >>
> >> Here's my new timeline:
> >>
> >> 18:14... - Stop Processors
> >> 18:15:40 - Shutdown NiFi (graceful and successful)
> >> 18:28:03 - Starting NiFi which seemingly deletes content
> >> 18:31++ - Add Funnel etc. and start Processors again (so only now do I
> see
> >> the problem occurring even though it probably would have happened
> without
> >> it as well)
> >>
> >> I've uploaded the relevant part of the log here <
> >> http://pastebin.com/6XWP5SVF>
> >>
> >> All processors involved are custom processors but they don't do anything
> >> special and have been running for days and survived multiple restarts
> >> already. I can't share code now but if it becomes important I can strip
> >> them to a bare minimum and share.
> >>
> >> So when the failure happened it was even easier: CustomSourceProcessor
> was
> >> connected to CustomDestinationProcessor via a normal connection.
> >>
> >> Thanks yet again for helping out everyone!
> >>
> >> On Wed, Feb 17, 2016 at 5:04 AM, Aldrin Piri 
> wrote:
> >>
> >>> Lars,
> >>>
> >>> Are you able to share your flow or a template of it so we can try to
> >>> recreate?
> >>>
> >>> If not, could you give some information as to what it is doing and what
> >>> processors/components are involved.  Are there any custom components?
> >>>
> >>> Thanks!
> >>>
> >>> On Tue, Feb 16, 2016 at 10:18 PM, Joe Witt  wrote:
> >>>
>  'that deletes the original file'
> 
>  True but even then that refers to the original source data and not
>  what it is in the content repository itself.  The content repository
>  error that was emitted about missing flow file exception/content not
>  found is for the purpose of signaling data was removed by some process
>  outside of NiFi.
> 
>  Mark Payne: Any ideas?
> 
>  On Tue, Feb 16, 2016 at 10:15 PM, Thad Guidry 
>  wrote:
> > There's a checkbox option in the FetchFile that deletes the original
>  file.
> >
> >
> 
> >>>
> https://github.com/apache/nifi/blob

[GitHub] nifi pull request: nifi-1514 - Fixed: ExecuteStreamCommand does no...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/223


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: nifi-1514 - Fixed: ExecuteStreamCommand does no...

[apiri]

Github user apiri commented on the pull request:

https://github.com/apache/nifi/pull/223#issuecomment-185443013
  
@michalklempa looks good.  one minor point that I am adjusting is the usage 
of newProperty at  
https://github.com/apache/nifi/pull/223/files#diff-fe8e954d730443b47cb0a94d340ac271R306
 This is typically for dynamic properties where new values must be created 
because there is no a priori knowledge of them.

Regardless, thanks for detecting and fixing this up and it will be merged 
shortly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1252

[pvillard31]

Github user pvillard31 commented on the pull request:

https://github.com/apache/nifi/pull/229#issuecomment-185406656
  
@apiri That's OK for me!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1252

[pvillard31]

Github user pvillard31 closed the pull request at:

https://github.com/apache/nifi/pull/229


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Set default yield duration

[Adam Lamar]

Bryan, Joe, Matt,

Thanks very much for your comments and creating a JIRA! It will 
definitely be useful for my use case.


Cheers,
Adam

On 2/17/16 11:46 AM, Bryan Bende wrote:

I created this JIRA:
https://issues.apache.org/jira/browse/NIFI-1526

-Bryan

On Wed, Feb 17, 2016 at 8:51 AM, Matthew Clarke 
wrote:


Every Processor's configuration should have 4 tabs (settings, scheduling,
properties, and comments). You can adjust the yield duration from the
settings tab and the run schedule from the scheduling tab.

Matt

On Wed, Feb 17, 2016 at 1:19 AM, Adam Lamar  wrote:


Hey everyone,

I've been working on a processor that yields often, and the default yield
duration of 1 second is too short. Is there any way to change this

default

for a specific processor only?

Alternatively, can the run schedule default duration be changed in a
similar way?

Cheers,
Adam

Re: FlowFiles missing (MissingFlowFileException)

[Mark Payne]

Lars,

Joe is correct - i felt a need for coffee before trying to identify the exact 
case
that will cause this issue to occur :)

I created a ticket for this issue here: 
https://issues.apache.org/jira/browse/NIFI-1527 


I share your opinion that the funnel is not related. Rather, I believe the 
issue has to do
with swapping of data. NiFi has a mechanism for swapping out FlowFiles to disk, 
rather
than leaving them all in memory (here we are talking about the FlowFile 
objects, which
contains things of attributes. The content is already written to disk and not 
kept in memory).

With the default settings, this happens if at least 30,000 FlowFiles exist in 
the same queue.
In this case, upon restart, if all of the FlowFiles that reference a specific 
"content file" on disk
are swapped out, the Content Repository will end up removing or archiving that 
data. If the
archive is disabled or full, the repository will end up removing it. I believe 
this is what you saw.
I believe the content was removed on restart because all FlowFiles that 
referenced it were
swapped out.

Please confirm what Joe asked below: that you either disabled archiving or that 
your disk is
at least 50% full (or that you changed that configuration parameter in 
conf/nifi.properties).

We will certainly be addressing this issue very promptly. A huge thank you for 
noticing the weirdness
and bringing it to our attention and for providing such great details!

-Mark


> On Feb 17, 2016, at 1:48 PM, Joe Witt  wrote:
> 
> Lars,
> 
> First of all thank you very much for reporting this and providing the
> detail you did.
> 
> Mark Payne just replicated the problem it sounds like then rather than
> emailing he decided to go get coffee :-).
> 
> We will be working this up for very prompt resolution and it warrants
> a release in my view.
> 
> Can you confirm that you are either not using archiving or you are
> using it and you have more than 50% of space on the partition nifi is
> running used up?
> 
> I'll let Mark share the details.
> 
> Thanks
> Joe
> 
> On Wed, Feb 17, 2016 at 4:06 AM, Lars Francke  wrote:
>> It seems as if the Funnel thing wasn't actually the problem.
>> 
>> Here's my new timeline:
>> 
>> 18:14... - Stop Processors
>> 18:15:40 - Shutdown NiFi (graceful and successful)
>> 18:28:03 - Starting NiFi which seemingly deletes content
>> 18:31++ - Add Funnel etc. and start Processors again (so only now do I see
>> the problem occurring even though it probably would have happened without
>> it as well)
>> 
>> I've uploaded the relevant part of the log here <
>> http://pastebin.com/6XWP5SVF>
>> 
>> All processors involved are custom processors but they don't do anything
>> special and have been running for days and survived multiple restarts
>> already. I can't share code now but if it becomes important I can strip
>> them to a bare minimum and share.
>> 
>> So when the failure happened it was even easier: CustomSourceProcessor was
>> connected to CustomDestinationProcessor via a normal connection.
>> 
>> Thanks yet again for helping out everyone!
>> 
>> On Wed, Feb 17, 2016 at 5:04 AM, Aldrin Piri  wrote:
>> 
>>> Lars,
>>> 
>>> Are you able to share your flow or a template of it so we can try to
>>> recreate?
>>> 
>>> If not, could you give some information as to what it is doing and what
>>> processors/components are involved.  Are there any custom components?
>>> 
>>> Thanks!
>>> 
>>> On Tue, Feb 16, 2016 at 10:18 PM, Joe Witt  wrote:
>>> 
 'that deletes the original file'
 
 True but even then that refers to the original source data and not
 what it is in the content repository itself.  The content repository
 error that was emitted about missing flow file exception/content not
 found is for the purpose of signaling data was removed by some process
 outside of NiFi.
 
 Mark Payne: Any ideas?
 
 On Tue, Feb 16, 2016 at 10:15 PM, Thad Guidry 
 wrote:
> There's a checkbox option in the FetchFile that deletes the original
 file.
> 
> 
 
>>> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/FetchFile.java#L62
> 
> static final AllowableValue COMPLETION_DELETE = new
 AllowableValue("Delete
> File", "Delete File", "Deletes the original file from the file
>>> system");
> 
> 
> Perhaps its something along those lines, maybe in his other processors
>>> ?
> He mentioned "I also added another processor feeding that same funnel"
 ...
> which processor was that exactly ?
> 
> 
> Thad
> +ThadGuidry 
> 
> On Tue, Feb 16, 2016 at 4:35 PM, Lars Francke 
> wrote:
> 
>> Thanks for the explanation.
>> 
>> I tried to reproduce but I can't. I also looked through my bash
>>> history
 and
>> I can't find anyth

[GitHub] nifi pull request: NIFI-1521 Allows use of SSL in AMQP Processor

[alopresto]

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/232#discussion_r53211145
  
--- Diff: 
nifi-nar-bundles/nifi-amqp-bundle/nifi-amqp-processors/src/main/java/org/apache/nifi/amqp/processors/AbstractAMQPProcessor.java
 ---
@@ -84,6 +89,72 @@
 .allowableValues("0.9.1")
 .defaultValue("0.9.1")
 .build();
+public static final PropertyDescriptor USE_SSL_PROTOCOL = new 
PropertyDescriptor.Builder()
+.name("Use SSL protocol")
+.description("Indicates whether or not SSL protocol should be 
used")
+.allowableValues("true", "false")
+.defaultValue("false")
+.required(true)
+.build();
+public static final PropertyDescriptor VALIDATE_CERTIFICATES = new 
PropertyDescriptor.Builder()
--- End diff --

Thanks @pvillard31 . I read the linked documentation, and what they are 
describing is if the trust chain (the cryptographic link between the presented 
certificate identifying a resource and a trusted certificate of a known entity 
proving the relationship) should be verified, specifically for _client 
certificates_. 

> When a web browser connects to an HTTPS web server, the server presents 
its public certificate, the web browser attempts to establish a chain of trust 
between the root certificates the browser is aware of and the server's 
certificate, and all being well, an encrypted communication channel is 
established. Although not used normally by web browsers and web servers, SSL 
allows the server to ask the client to present a certificate. In this way the 
server can verify that the client is who they say they are.

> This policy of whether or not the server asks for a certificate from the 
client, and whether or not they demand that they are able to trust the 
certificate, is what the verify and fail_if_no_peer_cert arguments control. 
Through the {fail_if_no_peer_cert,false} option, we state that we're prepared 
to accept clients which don't have a certificate to send us, but through the 
{verify,verify_peer} option, we state that if the client does send us a 
certificate, we must be able to establish a chain of trust to it. 

In this case, I am comfortable with not requiring the client to present a 
certificate, but I don't think we should provide the option to present an 
invalid/malformed certificate and successfully connect. If a client certificate 
is presented, we should require it to validate successfully against a 
certificate within the provided truststore. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: FlowFiles missing (MissingFlowFileException)

[Joe Witt]

Lars,

First of all thank you very much for reporting this and providing the
detail you did.

Mark Payne just replicated the problem it sounds like then rather than
emailing he decided to go get coffee :-).

We will be working this up for very prompt resolution and it warrants
a release in my view.

Can you confirm that you are either not using archiving or you are
using it and you have more than 50% of space on the partition nifi is
running used up?

I'll let Mark share the details.

Thanks
Joe

On Wed, Feb 17, 2016 at 4:06 AM, Lars Francke  wrote:
> It seems as if the Funnel thing wasn't actually the problem.
>
> Here's my new timeline:
>
> 18:14... - Stop Processors
> 18:15:40 - Shutdown NiFi (graceful and successful)
> 18:28:03 - Starting NiFi which seemingly deletes content
> 18:31++ - Add Funnel etc. and start Processors again (so only now do I see
> the problem occurring even though it probably would have happened without
> it as well)
>
> I've uploaded the relevant part of the log here <
> http://pastebin.com/6XWP5SVF>
>
> All processors involved are custom processors but they don't do anything
> special and have been running for days and survived multiple restarts
> already. I can't share code now but if it becomes important I can strip
> them to a bare minimum and share.
>
> So when the failure happened it was even easier: CustomSourceProcessor was
> connected to CustomDestinationProcessor via a normal connection.
>
> Thanks yet again for helping out everyone!
>
> On Wed, Feb 17, 2016 at 5:04 AM, Aldrin Piri  wrote:
>
>> Lars,
>>
>> Are you able to share your flow or a template of it so we can try to
>> recreate?
>>
>> If not, could you give some information as to what it is doing and what
>> processors/components are involved.  Are there any custom components?
>>
>> Thanks!
>>
>> On Tue, Feb 16, 2016 at 10:18 PM, Joe Witt  wrote:
>>
>> > 'that deletes the original file'
>> >
>> > True but even then that refers to the original source data and not
>> > what it is in the content repository itself.  The content repository
>> > error that was emitted about missing flow file exception/content not
>> > found is for the purpose of signaling data was removed by some process
>> > outside of NiFi.
>> >
>> > Mark Payne: Any ideas?
>> >
>> > On Tue, Feb 16, 2016 at 10:15 PM, Thad Guidry 
>> > wrote:
>> > > There's a checkbox option in the FetchFile that deletes the original
>> > file.
>> > >
>> > >
>> >
>> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/FetchFile.java#L62
>> > >
>> > > static final AllowableValue COMPLETION_DELETE = new
>> > AllowableValue("Delete
>> > > File", "Delete File", "Deletes the original file from the file
>> system");
>> > >
>> > >
>> > > Perhaps its something along those lines, maybe in his other processors
>> ?
>> > > He mentioned "I also added another processor feeding that same funnel"
>> > ...
>> > > which processor was that exactly ?
>> > >
>> > >
>> > > Thad
>> > > +ThadGuidry 
>> > >
>> > > On Tue, Feb 16, 2016 at 4:35 PM, Lars Francke 
>> > > wrote:
>> > >
>> > >> Thanks for the explanation.
>> > >>
>> > >> I tried to reproduce but I can't. I also looked through my bash
>> history
>> > and
>> > >> I can't find anything suspicious. I'm pretty sure that nothing deleted
>> > >> files in the content_repository that's not NiFi itself. Everything
>> else
>> > >> (logs etc.) are all untouched and some content files have survived as
>> > well.
>> > >> A few FlowFiles are being processed successfully and I just checked
>> the
>> > >> creation date of all files in content_repository. Most of them are
>> > "old".
>> > >>
>> > >> On Tue, Feb 16, 2016 at 11:12 PM, Joe Witt 
>> wrote:
>> > >>
>> > >> > Lars,
>> > >> >
>> > >> > The information you're providing from the logs is a pretty important
>> > >> > bit of debug data.
>> > >> >
>> > >> > This concept of 'CONTENTMISSING' being recorded into the Flow File
>> > >> > Repository is NiFI's way of saying "Hey I knew about this flow file
>> > >> > but when I tried to access the content it was no longer in the
>> content
>> > >> > repository".  What I'm suggesting is something outside of NiFi
>> itself
>> > >> > removed the content.  By default, even when you remove content using
>> > >> > the NiFi API it isn't actually deleting the content until it has to
>> > >> > and it is asynchronous.  Even if you had restarted NiFi during this
>> I
>> > >> > don't see how this could occur.
>> > >> >
>> > >> > Even if you have some bugs in the custom processor implementations
>> the
>> > >> > issue you're showing here should not be possible.
>> > >> >
>> > >> > The only explanation that makes sense to me so far is that the
>> content
>> > >> > was actually deleted from within the content repository by something
>> > >> > other than NiFi.
>> > >> >
>> > >> > Can you reproduce the issue?
>> > >> >
>> > >> > Tha

Re: Set default yield duration

[Bryan Bende]

I created this JIRA:
https://issues.apache.org/jira/browse/NIFI-1526

-Bryan

On Wed, Feb 17, 2016 at 8:51 AM, Matthew Clarke 
wrote:

> Every Processor's configuration should have 4 tabs (settings, scheduling,
> properties, and comments). You can adjust the yield duration from the
> settings tab and the run schedule from the scheduling tab.
>
> Matt
>
> On Wed, Feb 17, 2016 at 1:19 AM, Adam Lamar  wrote:
>
> > Hey everyone,
> >
> > I've been working on a processor that yields often, and the default yield
> > duration of 1 second is too short. Is there any way to change this
> default
> > for a specific processor only?
> >
> > Alternatively, can the run schedule default duration be changed in a
> > similar way?
> >
> > Cheers,
> > Adam
> >
>

Re: Set default yield duration

[Matthew Clarke]

Every Processor's configuration should have 4 tabs (settings, scheduling,
properties, and comments). You can adjust the yield duration from the
settings tab and the run schedule from the scheduling tab.

Matt

On Wed, Feb 17, 2016 at 1:19 AM, Adam Lamar  wrote:

> Hey everyone,
>
> I've been working on a processor that yields often, and the default yield
> duration of 1 second is too short. Is there any way to change this default
> for a specific processor only?
>
> Alternatively, can the run schedule default duration be changed in a
> similar way?
>
> Cheers,
> Adam
>

Re: FlowFiles missing (MissingFlowFileException)

[Lars Francke]

It seems as if the Funnel thing wasn't actually the problem.

Here's my new timeline:

18:14... - Stop Processors
18:15:40 - Shutdown NiFi (graceful and successful)
18:28:03 - Starting NiFi which seemingly deletes content
18:31++ - Add Funnel etc. and start Processors again (so only now do I see
the problem occurring even though it probably would have happened without
it as well)

I've uploaded the relevant part of the log here <
http://pastebin.com/6XWP5SVF>

All processors involved are custom processors but they don't do anything
special and have been running for days and survived multiple restarts
already. I can't share code now but if it becomes important I can strip
them to a bare minimum and share.

So when the failure happened it was even easier: CustomSourceProcessor was
connected to CustomDestinationProcessor via a normal connection.

Thanks yet again for helping out everyone!

On Wed, Feb 17, 2016 at 5:04 AM, Aldrin Piri  wrote:

> Lars,
>
> Are you able to share your flow or a template of it so we can try to
> recreate?
>
> If not, could you give some information as to what it is doing and what
> processors/components are involved.  Are there any custom components?
>
> Thanks!
>
> On Tue, Feb 16, 2016 at 10:18 PM, Joe Witt  wrote:
>
> > 'that deletes the original file'
> >
> > True but even then that refers to the original source data and not
> > what it is in the content repository itself.  The content repository
> > error that was emitted about missing flow file exception/content not
> > found is for the purpose of signaling data was removed by some process
> > outside of NiFi.
> >
> > Mark Payne: Any ideas?
> >
> > On Tue, Feb 16, 2016 at 10:15 PM, Thad Guidry 
> > wrote:
> > > There's a checkbox option in the FetchFile that deletes the original
> > file.
> > >
> > >
> >
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/FetchFile.java#L62
> > >
> > > static final AllowableValue COMPLETION_DELETE = new
> > AllowableValue("Delete
> > > File", "Delete File", "Deletes the original file from the file
> system");
> > >
> > >
> > > Perhaps its something along those lines, maybe in his other processors
> ?
> > > He mentioned "I also added another processor feeding that same funnel"
> > ...
> > > which processor was that exactly ?
> > >
> > >
> > > Thad
> > > +ThadGuidry 
> > >
> > > On Tue, Feb 16, 2016 at 4:35 PM, Lars Francke 
> > > wrote:
> > >
> > >> Thanks for the explanation.
> > >>
> > >> I tried to reproduce but I can't. I also looked through my bash
> history
> > and
> > >> I can't find anything suspicious. I'm pretty sure that nothing deleted
> > >> files in the content_repository that's not NiFi itself. Everything
> else
> > >> (logs etc.) are all untouched and some content files have survived as
> > well.
> > >> A few FlowFiles are being processed successfully and I just checked
> the
> > >> creation date of all files in content_repository. Most of them are
> > "old".
> > >>
> > >> On Tue, Feb 16, 2016 at 11:12 PM, Joe Witt 
> wrote:
> > >>
> > >> > Lars,
> > >> >
> > >> > The information you're providing from the logs is a pretty important
> > >> > bit of debug data.
> > >> >
> > >> > This concept of 'CONTENTMISSING' being recorded into the Flow File
> > >> > Repository is NiFI's way of saying "Hey I knew about this flow file
> > >> > but when I tried to access the content it was no longer in the
> content
> > >> > repository".  What I'm suggesting is something outside of NiFi
> itself
> > >> > removed the content.  By default, even when you remove content using
> > >> > the NiFi API it isn't actually deleting the content until it has to
> > >> > and it is asynchronous.  Even if you had restarted NiFi during this
> I
> > >> > don't see how this could occur.
> > >> >
> > >> > Even if you have some bugs in the custom processor implementations
> the
> > >> > issue you're showing here should not be possible.
> > >> >
> > >> > The only explanation that makes sense to me so far is that the
> content
> > >> > was actually deleted from within the content repository by something
> > >> > other than NiFi.
> > >> >
> > >> > Can you reproduce the issue?
> > >> >
> > >> > Thanks
> > >> > Joe
> > >> >
> > >> > On Tue, Feb 16, 2016 at 4:58 PM, Lars Francke <
> lars.fran...@gmail.com
> > >
> > >> > wrote:
> > >> > > Any ideas on how to debug this further?
> > >> > >
> > >> > > I know very little about the internals of NiFi but there are
> > obviously
> > >> > > still references to that content and it shouldn't have been
> deleted.
> > >> Can
> > >> > > you think of a way I could have done this by accident?
> > >> > >
> > >> > > On Tue, Feb 16, 2016 at 10:35 PM, Joe Witt 
> > wrote:
> > >> > >
> > >> > >> Interesting.  What that suggests is the content has been removed
> > from
> > >> > >> the content repo itself.
> > >> > >>
> > >> > >> Thanks
> > >> > >> Joe
> 

Re: Set default yield duration

[Joe Witt]

I think it is a good idea to allow a processor to establish what are
reasonable defaults for itself then let users tune it as they can
today.  Definitely a good JIRA to fire up.

On Wed, Feb 17, 2016 at 9:30 AM, Bryan Bende  wrote:
> Adam,
>
> I think the idea was for the user to always control these values through
> the user interface, and therefore there isn't a way that I know of for a
> specific processor to control the default values for yield duration and run
> schedule.
>
> I have always wanted to be able to do this though, specifically for the run
> schedule. The main scenario is a "Get" processor that is going to poll an
> external system and extract data. In this case, a run schedule of 0 seconds
> is almost never the desired value, and if a user runs the processor
> forgetting to change the duration, they start killing the external system.
>
> Curious to hear what others think.
>
> -Bryan
>
>
> On Wed, Feb 17, 2016 at 1:19 AM, Adam Lamar  wrote:
>
>> Hey everyone,
>>
>> I've been working on a processor that yields often, and the default yield
>> duration of 1 second is too short. Is there any way to change this default
>> for a specific processor only?
>>
>> Alternatively, can the run schedule default duration be changed in a
>> similar way?
>>
>> Cheers,
>> Adam
>>

Re: Set default yield duration

[Bryan Bende]

Adam,

I think the idea was for the user to always control these values through
the user interface, and therefore there isn't a way that I know of for a
specific processor to control the default values for yield duration and run
schedule.

I have always wanted to be able to do this though, specifically for the run
schedule. The main scenario is a "Get" processor that is going to poll an
external system and extract data. In this case, a run schedule of 0 seconds
is almost never the desired value, and if a user runs the processor
forgetting to change the duration, they start killing the external system.

Curious to hear what others think.

-Bryan


On Wed, Feb 17, 2016 at 1:19 AM, Adam Lamar  wrote:

> Hey everyone,
>
> I've been working on a processor that yields often, and the default yield
> duration of 1 second is too short. Is there any way to change this default
> for a specific processor only?
>
> Alternatively, can the run schedule default duration be changed in a
> similar way?
>
> Cheers,
> Adam
>

[GitHub] nifi pull request: NIFI-1521 Allows use of SSL in AMQP Processor

[pvillard31]

Github user pvillard31 commented on the pull request:

https://github.com/apache/nifi/pull/232#issuecomment-185326467
  
Yes, you're right... I didn't know about this feature. I will change the 
processor to use the SSL Context as suggested.

@olegz Regarding unit testing, I am not sure to picture what you want to 
achieve with a unit test. We could mock up the ConnectionFactory as well but 
for what purpose? I had a look at the [RabbitMQ unit tests regarding 
SSL](https://github.com/rabbitmq/rabbitmq-java-client/tree/master/src/test/java/com/rabbitmq/client/test/ssl)
 part and it seems to be relying on specific set-up/tools.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: nifi question

[Joe Witt]

Hello,

Could you share a screenshot.  Also, which web browser (and version)
are you using ?  Finally, which version of NiFi?

Thanks
JOe

On Wed, Feb 17, 2016 at 10:24 AM, yonatan kebede  wrote:
> hi,
> i am new to nifi and have got problems on the UI, that its not displaying
> the icons on top of the processors and also in the connections/relations,
> it was working in windows 8 but when i update it to win 10 , sth like this
> happen?
>
> what do you think is the problem? or any helpis appreciated
>
> Regards,
>
> Yonatan Kebede.

nifi question

[yonatan kebede]

hi,
i am new to nifi and have got problems on the UI, that its not displaying
the icons on top of the processors and also in the connections/relations,
it was working in windows 8 but when i update it to win 10 , sth like this
happen?

what do you think is the problem? or any helpis appreciated

Regards,

Yonatan Kebede.

[GitHub] nifi pull request: NIFI-1521 Allows use of SSL in AMQP Processor

[joewitt]

Github user joewitt commented on the pull request:

https://github.com/apache/nifi/pull/232#issuecomment-185323176
  
+1 to Bryan's point.  We should definitely try to leverage the SSL Context. 
 Oleg is this something you can work with Pierre on with the unit testing?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1521 Allows use of SSL in AMQP Processor

[bbende]

Github user bbende commented on the pull request:

https://github.com/apache/nifi/pull/232#issuecomment-185322119
  
Have we considered using the SSLContextService here rather than specifying 
all of the keystore/truststore properties in the processor?

Usually the processor will have logic that says "if no SSLContextService 
was selected then we are doing regular communication, if one is selected then 
secure communication". This way you can avoid another property to specify if 
SSL/TLS should be used.

An example of using this service is in InvokeHTTP...

Defining the property:

https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java#L196

Using the service:

https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java#L479


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1252

[apiri]

Github user apiri commented on the pull request:

https://github.com/apache/nifi/pull/229#issuecomment-185321319
  
@pvillard31 Thank you for contributing this fix against the last release.  
This was handled via NIFI-1415 in commit 
93e2c5219599f24b54517dd43195c3d121030c5e as part of the 0.5.0 release.  I was 
not able to reproduce and believe it should be have as desired.  Please let us 
know if this is not the case.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1420 Adding Splunk bundle

[bbende]

GitHub user bbende opened a pull request:

https://github.com/apache/nifi/pull/233

NIFI-1420 Adding Splunk bundle 

## Overview
This pull request adds a Splunk bundle with the following processors:
 
* **ListenSplunkForwarder** - Receives data from a Splunk forwarder, based 
off the AbstractListenEventProcessor created during RELP work.

* **PutSplunk** - Delivers to Splunk Enterprise over TCP or UDP, based some 
of the design off PutKafka for how to handle delimited messages, also created 
an AbstractPutEventProcessor to  make future "Put" TCP/UDP processors easier, 
involved refactoring code from PutSyslog.

* **GetSplunk** - Extracts data from Splunk Enterprise based on a query, 
processor provides ability to optionally specify a time range, or have the 
processor manage the time ranges using the new state management API.

## Testing

### Download and extract the forwarder and Splunk enterprise:
http://www.splunk.com/en_us/download/universal-forwarder.html
http://www.splunk.com/en_us/download/splunk-enterprise.html

## Splunk Forwarder:

Edit/Create splunkforwarder/etc/system/local/outputs.conf and configure an 
output for NiFi to listen to:

[tcpout:nifi]
server=localhost:6588
sendCookedData=false

Start the forwarder:

./splunkforwarder/bin/splunk start

At this point you can create a NiFi flow with ListenSplunkForwarder, 
listening on TCP port 6588 and should be receiving data.

## Splunk Enterprise

Start Splunk Enterprise

./splunk/bin/splunk start

Go to http://localhost:8000 in your browser

Create an input from the Settings -> Data Inputs menu, create a TCP or UDP 
input.

After that you should be able to use PutSplunk to deliver data to the input 
created in the previous step, and GetSplunk to extract data. 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/bbende/nifi NIFI-1420

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/233.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #233






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1521 Allows use of SSL in AMQP Processor

[pvillard31]

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/232#discussion_r53195449
  
--- Diff: 
nifi-nar-bundles/nifi-amqp-bundle/nifi-amqp-processors/src/main/java/org/apache/nifi/amqp/processors/AbstractAMQPProcessor.java
 ---
@@ -84,6 +89,72 @@
 .allowableValues("0.9.1")
 .defaultValue("0.9.1")
 .build();
+public static final PropertyDescriptor USE_SSL_PROTOCOL = new 
PropertyDescriptor.Builder()
+.name("Use SSL protocol")
+.description("Indicates whether or not SSL protocol should be 
used")
+.allowableValues("true", "false")
+.defaultValue("false")
+.required(true)
+.build();
+public static final PropertyDescriptor VALIDATE_CERTIFICATES = new 
PropertyDescriptor.Builder()
--- End diff --

@alopresto In fact I added this possibility because this is a use case 
described in [the RabbitMQ documentation](https://www.rabbitmq.com/ssl.html), 
and I assumed this is a possible scenario. Regarding the default value, I 
definitely agree with you, my bad.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1521 Allows use of SSL in AMQP Processor

[alopresto]

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/232#discussion_r53186223
  
--- Diff: 
nifi-nar-bundles/nifi-amqp-bundle/nifi-amqp-processors/src/main/java/org/apache/nifi/amqp/processors/AbstractAMQPProcessor.java
 ---
@@ -84,6 +89,72 @@
 .allowableValues("0.9.1")
 .defaultValue("0.9.1")
 .build();
+public static final PropertyDescriptor USE_SSL_PROTOCOL = new 
PropertyDescriptor.Builder()
+.name("Use SSL protocol")
+.description("Indicates whether or not SSL protocol should be 
used")
+.allowableValues("true", "false")
+.defaultValue("false")
+.required(true)
+.build();
+public static final PropertyDescriptor VALIDATE_CERTIFICATES = new 
PropertyDescriptor.Builder()
--- End diff --

I am not sure why we would allow certificates _not_ be validated on a 
secure connection. I recommend removing this option, and I strongly oppose to 
the default value being `false` if it is allowed to remain. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1521 Allows use of SSL in AMQP Processor

[olegz]

Github user olegz commented on the pull request:

https://github.com/apache/nifi/pull/232#issuecomment-185272014
  
Ok, let's collaborate. If you can simply describe what you needed to do on 
both client and server to have SSL, I can definitely mock it on the 
_TestConnection_ and _TestChannel_.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1521 Allows use of SSL in AMQP Processor

[pvillard31]

Github user pvillard31 commented on the pull request:

https://github.com/apache/nifi/pull/232#issuecomment-185270244
  
Well I must admit I locally tested the change with a RabbitMQ server, 
but that's all. I had a look at the existing unit tests for AMQP processors but 
I am not sure how to mock a SSL context. If you have some hints I'll do my best 
to add unit tests ;)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1521 Allows use of SSL in AMQP Processor

[olegz]

Github user olegz commented on the pull request:

https://github.com/apache/nifi/pull/232#issuecomment-185265232
  
Pierre, first of all, huge thanks for the contribution! Quite an important 
feature especially for enterprise users. Quick question though. One of the 
initiatives we are going through at the moment is to improve test coverage of 
our components. I know it's not trivial to the infrastructure tests for things 
like SSL, but I still wanted to ask if you thought about it and if any type of 
tests you ran locally could be migrated into unit tests even if mocks has to be 
used. I can also help with that as well as I am a bit paranoid when it comes to 
unit testing ;)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1521 Allows use of SSL in AMQP Processor

[pvillard31]

GitHub user pvillard31 opened a pull request:

https://github.com/apache/nifi/pull/232

NIFI-1521 Allows use of SSL in AMQP Processor

Allows use of SSL in AMQP Processor

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pvillard31/nifi NIFI-1521

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/232.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #232


commit 4ae03bb7ed6b2739351bcda58528d16b497821ff
Author: Pierre Villard 
Date:   2016-02-16T10:02:10Z

Merge remote-tracking branch 'refs/remotes/apache/master'

commit e3c6b5741b22d22a4c9a3476c843be5390171353
Author: Pierre Villard 
Date:   2016-02-17T10:39:01Z

Merge remote-tracking branch 'refs/remotes/apache/master'

commit 241c11a914dcea2a01f7e74732e40e511685122c
Author: Pierre Villard 
Date:   2016-02-17T15:30:09Z

NIFI-1521 Allows use of SSL in AMQP Processor

Allows use of SSL in AMQP Processor




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1519 upgraded to the latest version of Spa...

[olegz]

GitHub user olegz opened a pull request:

https://github.com/apache/nifi/pull/231

NIFI-1519 upgraded to the latest version of Spark 1.6.0



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/olegz/nifi NIFI-1519

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/231.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #231


commit 9ee4404640ee5f8413a455c666983ae7f47199a3
Author: Oleg Zhurakousky 
Date:   2016-02-17T15:29:24Z

NIFI-1519 upgraded to the latest version of Spark 1.6.0




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: Fix NiFi-1501 : Test Monitor Activity has spuri...

[joewitt]

Github user joewitt commented on the pull request:

https://github.com/apache/nifi/pull/225#issuecomment-185230292
  
@PuspenduBanerjee i will add you now as a contributor to the apache nifi 
wiki so you can do so.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1464, NIFI-78, NIFI-1487 Refactored Proces...

[joewitt]

Github user joewitt commented on a diff in the pull request:

https://github.com/apache/nifi/pull/210#discussion_r53169197
  
--- Diff: 
nifi-commons/nifi-properties/src/main/java/org/apache/nifi/util/NiFiProperties.java
 ---
@@ -71,6 +71,7 @@
 public static final String ADMINISTRATIVE_YIELD_DURATION = 
"nifi.administrative.yield.duration";
 public static final String PERSISTENT_STATE_DIRECTORY = 
"nifi.persistent.state.directory";
 public static final String BORED_YIELD_DURATION = 
"nifi.bored.yield.duration";
+public static final String PROCESSOR_START_TIMEOUT = 
"nifi.processor.start.timeout";
--- End diff --

I am definitely on the same page with you there  
https://issues.apache.org/jira/browse/NIFI-1522


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1464, NIFI-78, NIFI-1487 Refactored Proces...

[olegz]

Github user olegz commented on a diff in the pull request:

https://github.com/apache/nifi/pull/210#discussion_r53169078
  
--- Diff: 
nifi-commons/nifi-properties/src/main/java/org/apache/nifi/util/NiFiProperties.java
 ---
@@ -71,6 +71,7 @@
 public static final String ADMINISTRATIVE_YIELD_DURATION = 
"nifi.administrative.yield.duration";
 public static final String PERSISTENT_STATE_DIRECTORY = 
"nifi.persistent.state.directory";
 public static final String BORED_YIELD_DURATION = 
"nifi.bored.yield.duration";
+public static final String PROCESSOR_START_TIMEOUT = 
"nifi.processor.start.timeout";
--- End diff --

One last comment. In any event I think we should still support a simple 
expression of time in numbers treating them as milliseconds. For example if 
time configuration value comes as '1', then it should be treated as 1 
millisecond. At least this way we can argue both points and see what users use 
most.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1464, NIFI-78, NIFI-1487 Refactored Proces...

[joewitt]

Github user joewitt commented on a diff in the pull request:

https://github.com/apache/nifi/pull/210#discussion_r53167209
  
--- Diff: 
nifi-commons/nifi-properties/src/main/java/org/apache/nifi/util/NiFiProperties.java
 ---
@@ -71,6 +71,7 @@
 public static final String ADMINISTRATIVE_YIELD_DURATION = 
"nifi.administrative.yield.duration";
 public static final String PERSISTENT_STATE_DIRECTORY = 
"nifi.persistent.state.directory";
 public static final String BORED_YIELD_DURATION = 
"nifi.bored.yield.duration";
+public static final String PROCESSOR_START_TIMEOUT = 
"nifi.processor.start.timeout";
--- End diff --

I can see why convention over configuration is strong within a development 
community but what we observed across a wide variety of a user community was 
very different need.

Regarding what is arguably 'natural' from one language to another this is a 
great point that I personally had not considered.  Fortunately I believe the 
set of supported terms to accept as indicators of periods can be substantially 
expanded though I am not aware of how problematic ambiguity could be.  We could 
of course choose to support some specific standard for expression of time 
periods though so still have viable paths.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1464, NIFI-78, NIFI-1487 Refactored Proces...

[olegz]

Github user olegz commented on a diff in the pull request:

https://github.com/apache/nifi/pull/210#discussion_r53166575
  
--- Diff: 
nifi-commons/nifi-properties/src/main/java/org/apache/nifi/util/NiFiProperties.java
 ---
@@ -71,6 +71,7 @@
 public static final String ADMINISTRATIVE_YIELD_DURATION = 
"nifi.administrative.yield.duration";
 public static final String PERSISTENT_STATE_DIRECTORY = 
"nifi.persistent.state.directory";
 public static final String BORED_YIELD_DURATION = 
"nifi.bored.yield.duration";
+public static final String PROCESSOR_START_TIMEOUT = 
"nifi.processor.start.timeout";
--- End diff --

Very true and unfortunately this is where I would go back to the 
"convention-over-configuration" discussion where if we declare that 
MILLISECONDS is a convention (the most widely used, hence natural and 
intuitive) for expression of all things time related then consistency would 
come. 
I am just afraid that we are actually asking user to do more (think more), 
rather then less. On top of that it creates more possibilities of spelling 
errors to deal with in the event user describes it in a way we can't parse, 
which may come with international users who I presume may naturally type things 
in their own language variation ('seconde', 'segundo' etc.)

Anyway, I've changed it, so we're good here, but would like this discussion 
to continue at some point may be on the WIKI or dev list.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1464, NIFI-78, NIFI-1487 Refactored Proces...

[joewitt]

Github user joewitt commented on a diff in the pull request:

https://github.com/apache/nifi/pull/210#discussion_r53165063
  
--- Diff: 
nifi-commons/nifi-properties/src/main/java/org/apache/nifi/util/NiFiProperties.java
 ---
@@ -71,6 +71,7 @@
 public static final String ADMINISTRATIVE_YIELD_DURATION = 
"nifi.administrative.yield.duration";
 public static final String PERSISTENT_STATE_DIRECTORY = 
"nifi.persistent.state.directory";
 public static final String BORED_YIELD_DURATION = 
"nifi.bored.yield.duration";
+public static final String PROCESSOR_START_TIMEOUT = 
"nifi.processor.start.timeout";
--- End diff --

regarding user friendliness the issue we saw before providing a utility for 
more natural expression of duration is that people were very inconsistent in 
specifying the unit of time that a given value applied to and using at times 
periods which didn't make sense for the scenario.  This model allows the user 
to express in a much more natural way both the period of time and unit of time.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1464, NIFI-78, NIFI-1487 Refactored Proces...

[olegz]

Github user olegz commented on a diff in the pull request:

https://github.com/apache/nifi/pull/210#discussion_r53164352
  
--- Diff: 
nifi-commons/nifi-properties/src/main/java/org/apache/nifi/util/NiFiProperties.java
 ---
@@ -71,6 +71,7 @@
 public static final String ADMINISTRATIVE_YIELD_DURATION = 
"nifi.administrative.yield.duration";
 public static final String PERSISTENT_STATE_DIRECTORY = 
"nifi.persistent.state.directory";
 public static final String BORED_YIELD_DURATION = 
"nifi.bored.yield.duration";
+public static final String PROCESSOR_START_TIMEOUT = 
"nifi.processor.start.timeout";
--- End diff --

I don't think we need to add default since it's handled during retrieval 
(basically if property is null or doesn't exist). 
I'll change to string based representation of time although I am still 
struggling with the whole concept and if indeed it is more user friendly. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: Enhancement for NIFI-1045 : Add "backup suffix"...

[trixpan]

Github user trixpan commented on the pull request:

https://github.com/apache/nifi/pull/230#issuecomment-185094912
  
Wouldn't it be a better choice to change the name of the new file instead 
of the existent one? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---