[jira] [Resolved] (NIFI-5433) Global Menu Doesn't open on Chrome
[ https://issues.apache.org/jira/browse/NIFI-5433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Jackson resolved NIFI-5433. Resolution: Resolved > Global Menu Doesn't open on Chrome > -- > > Key: NIFI-5433 > URL: https://issues.apache.org/jira/browse/NIFI-5433 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.7.0 >Reporter: Mark Jackson >Priority: Minor > > After installing nifi 1.7.0, you start up nifi (no changes). Then you click > the global menu at the top right (the kebab menu; aka the one with the 3 > horizontal lines) and nothing happens. There are no messages in the console > or any network requests. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5433) Global Menu Doesn't open on Chrome
[ https://issues.apache.org/jira/browse/NIFI-5433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16547324#comment-16547324 ] Mark Jackson commented on NIFI-5433: Thanks! That solved it. > Global Menu Doesn't open on Chrome > -- > > Key: NIFI-5433 > URL: https://issues.apache.org/jira/browse/NIFI-5433 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.7.0 >Reporter: Mark Jackson >Priority: Minor > > After installing nifi 1.7.0, you start up nifi (no changes). Then you click > the global menu at the top right (the kebab menu; aka the one with the 3 > horizontal lines) and nothing happens. There are no messages in the console > or any network requests. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5437) Yield in ExecuteScript when catching ScriptException
[
https://issues.apache.org/jira/browse/NIFI-5437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mike Thomsen updated NIFI-5437:
---
Resolution: Fixed
Fix Version/s: 1.8.0
Status: Resolved (was: Patch Available)
> Yield in ExecuteScript when catching ScriptException
>
>
> Key: NIFI-5437
> URL: https://issues.apache.org/jira/browse/NIFI-5437
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Major
> Fix For: 1.8.0
>
>
> In case of ScriptException in ExecuteScript, a flow file can be processed
> again and again which can lead to resource exhaustion and node instability.
> Proposition: yield the processor when an exception is caught before throwing
> a Process Exception.
> Example of exception:
> {code:java}
> 2018-07-17 16:15:23,392 ERROR [Timer-Driven Process Thread-9]
> o.a.nifi.processors.script.ExecuteScript
> ExecuteScript[id=9d776ec4-e9d7-1bca--be6c5019] Failed to process
> session due to org.apache.nifi.processor.exception.ProcessException:
> javax.script.ScriptException: TypeError: unsupported operand type(s) for +:
> 'int' and 'str' in
[jira] [Commented] (NIFI-5437) Yield in ExecuteScript when catching ScriptException
[
https://issues.apache.org/jira/browse/NIFI-5437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16547242#comment-16547242
]
ASF GitHub Bot commented on NIFI-5437:
--
Github user asfgit closed the pull request at:
https://github.com/apache/nifi/pull/2903
> Yield in ExecuteScript when catching ScriptException
>
>
> Key: NIFI-5437
> URL: https://issues.apache.org/jira/browse/NIFI-5437
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Major
>
> In case of ScriptException in ExecuteScript, a flow file can be processed
> again and again which can lead to resource exhaustion and node instability.
> Proposition: yield the processor when an exception is caught before throwing
> a Process Exception.
> Example of exception:
> {code:java}
> 2018-07-17 16:15:23,392 ERROR [Timer-Driven Process Thread-9]
> o.a.nifi.processors.script.ExecuteScript
> ExecuteScript[id=9d776ec4-e9d7-1bca--be6c5019] Failed to process
> session due to org.apache.nifi.processor.exception.ProcessException:
> javax.script.ScriptException: TypeError: unsupported operand type(s) for +:
> 'int' and 'str' in
[GitHub] nifi pull request #2903: NIFI-5437 - Yield in ExecuteScript when catching Sc...
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/2903 ---
[jira] [Commented] (NIFI-5437) Yield in ExecuteScript when catching ScriptException
[
https://issues.apache.org/jira/browse/NIFI-5437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16547240#comment-16547240
]
ASF subversion and git services commented on NIFI-5437:
---
Commit d126743d93f6d754774b0d3074f3464e01430eda in nifi's branch
refs/heads/master from [~pvillard]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=d126743 ]
NIFI-5437 - Yield in ExecuteScript when catching ScriptException
This closes #2903
Signed-off-by: Mike Thomsen
> Yield in ExecuteScript when catching ScriptException
>
>
> Key: NIFI-5437
> URL: https://issues.apache.org/jira/browse/NIFI-5437
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Major
>
> In case of ScriptException in ExecuteScript, a flow file can be processed
> again and again which can lead to resource exhaustion and node instability.
> Proposition: yield the processor when an exception is caught before throwing
> a Process Exception.
> Example of exception:
> {code:java}
> 2018-07-17 16:15:23,392 ERROR [Timer-Driven Process Thread-9]
> o.a.nifi.processors.script.ExecuteScript
> ExecuteScript[id=9d776ec4-e9d7-1bca--be6c5019] Failed to process
> session due to org.apache.nifi.processor.exception.ProcessException:
> javax.script.ScriptException: TypeError: unsupported operand type(s) for +:
> 'int' and 'str' in
[jira] [Resolved] (NIFI-5414) Release Apache NiFi 1.7.1
[ https://issues.apache.org/jira/browse/NIFI-5414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andy LoPresto resolved NIFI-5414. - Resolution: Done > Release Apache NiFi 1.7.1 > - > > Key: NIFI-5414 > URL: https://issues.apache.org/jira/browse/NIFI-5414 > Project: Apache NiFi > Issue Type: Task > Components: Tools and Build >Affects Versions: 1.7.1 >Reporter: Andy LoPresto >Assignee: Andy LoPresto >Priority: Blocker > Labels: release > Fix For: 1.7.1 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5196) AbstractJMSProcessor can leave connection hanging open
[
https://issues.apache.org/jira/browse/NIFI-5196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16547154#comment-16547154
]
ASF GitHub Bot commented on NIFI-5196:
--
GitHub user mosermw opened a pull request:
https://github.com/apache/nifi/pull/2904
NIFI-5196 Fixed JMS connection leak when uncaught exceptions happen
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mosermw/nifi NIFI-5196
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/2904.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2904
commit 4d4ef545448223cf5cd9bceea1b7778fab535480
Author: Mike Moser
Date: 2018-07-17T21:20:47Z
NIFI-5196 Fixed JMS connection leak when uncaught exceptions happen
> AbstractJMSProcessor can leave connection hanging open
> --
>
> Key: NIFI-5196
> URL: https://issues.apache.org/jira/browse/NIFI-5196
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.6.0
>Reporter: Nick Coleman
>Assignee: Sivaprasanna Sethuraman
>Priority: Major
> Labels: JMS
>
> ConsumeJMS and PublishJMS are based on AbstractJMSProcessor. They can cause
> a connection to the MQ Server to be opened and not be closed until the NiFi
> server is rebooted.
> This can create a problem for an MQ when the initial setup entered is wrong
> for an IBM MQ system that only allows one connection per user. Subsequent
> connections are blocked as the first remains open. Another potential problem
> even if the subsequent connection works is the original connection is still
> open and taking up resources.
> A simple change to the AbstractJMSProcessor would be in the onTrigger()
> function:
>
> {code:java}
> @Override
> public void onTrigger(ProcessContext context, ProcessSession session) throws
> ProcessException {
> T worker = workerPool.poll();
> if (worker == null) {
> worker = buildTargetResource(context);
> }
> boolean offered = false;
> try {
> rendezvousWithJms(context, session, worker);
> offered = workerPool.offer(worker);
> }
> finally {
> if (!offered) {
> worker.shutdown();
> }
> }
> }{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #2904: NIFI-5196 Fixed JMS connection leak when uncaught e...
GitHub user mosermw opened a pull request: https://github.com/apache/nifi/pull/2904 NIFI-5196 Fixed JMS connection leak when uncaught exceptions happen You can merge this pull request into a Git repository by running: $ git pull https://github.com/mosermw/nifi NIFI-5196 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2904.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2904 commit 4d4ef545448223cf5cd9bceea1b7778fab535480 Author: Mike Moser Date: 2018-07-17T21:20:47Z NIFI-5196 Fixed JMS connection leak when uncaught exceptions happen ---
[jira] [Commented] (NIFI-5432) Add Syslog Record Reader legacy Syslog
[ https://issues.apache.org/jira/browse/NIFI-5432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16547152#comment-16547152 ] ASF GitHub Bot commented on NIFI-5432: -- Github user ottobackwards commented on the issue: https://github.com/apache/nifi/pull/2900 Thanks for the review @bbende > Add Syslog Record Reader legacy Syslog > -- > > Key: NIFI-5432 > URL: https://issues.apache.org/jira/browse/NIFI-5432 > Project: Apache NiFi > Issue Type: Sub-task >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > Fix For: 1.8.0 > > > Create a record reader, based on nifi-syslog-utils for reading syslog without > support for structured data -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5436) TestRunner should allow setting the processor name
[ https://issues.apache.org/jira/browse/NIFI-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16547153#comment-16547153 ] ASF GitHub Bot commented on NIFI-5436: -- Github user ottobackwards commented on the issue: https://github.com/apache/nifi/pull/2902 Thanks for the review @alopresto > TestRunner should allow setting the processor name > -- > > Key: NIFI-5436 > URL: https://issues.apache.org/jira/browse/NIFI-5436 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > > Currently, the ProcessContext.getName() just returns a blank String. Some > users use that value of the context, but cannot test their processors > properly because there is no way to set this value. > > The TestRunner down to the MockProcessContext should support setting the name > of the processor when you create the TestRunner instance. > > If no name is provided, the the Processor class name should be used. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2902: NIFI-5436 add ability to set the processor name in the Pro...
Github user ottobackwards commented on the issue: https://github.com/apache/nifi/pull/2902 Thanks for the review @alopresto ---
[GitHub] nifi issue #2900: NIFI-5432 Add Syslog Record Reader legacy Syslog
Github user ottobackwards commented on the issue: https://github.com/apache/nifi/pull/2900 Thanks for the review @bbende ---
[jira] [Commented] (NIFI-5436) TestRunner should allow setting the processor name
[ https://issues.apache.org/jira/browse/NIFI-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16547139#comment-16547139 ] ASF GitHub Bot commented on NIFI-5436: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/2902 > TestRunner should allow setting the processor name > -- > > Key: NIFI-5436 > URL: https://issues.apache.org/jira/browse/NIFI-5436 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > > Currently, the ProcessContext.getName() just returns a blank String. Some > users use that value of the context, but cannot test their processors > properly because there is no way to set this value. > > The TestRunner down to the MockProcessContext should support setting the name > of the processor when you create the TestRunner instance. > > If no name is provided, the the Processor class name should be used. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #2902: NIFI-5436 add ability to set the processor name in ...
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/2902 ---
[jira] [Commented] (NIFI-5436) TestRunner should allow setting the processor name
[ https://issues.apache.org/jira/browse/NIFI-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16547138#comment-16547138 ] ASF subversion and git services commented on NIFI-5436: --- Commit 3746ae258d619c38d763c580f1acb6521c6c3bc5 in nifi's branch refs/heads/master from [~ottobackwards] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=3746ae2 ] NIFI-5436 Add ability to set the processor name in the ProcessContext when using the TestRunner. This closes #2902. Signed-off-by: Andy LoPresto > TestRunner should allow setting the processor name > -- > > Key: NIFI-5436 > URL: https://issues.apache.org/jira/browse/NIFI-5436 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > > Currently, the ProcessContext.getName() just returns a blank String. Some > users use that value of the context, but cannot test their processors > properly because there is no way to set this value. > > The TestRunner down to the MockProcessContext should support setting the name > of the processor when you create the TestRunner instance. > > If no name is provided, the the Processor class name should be used. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5139) ListenSyslog should process Structured Data
[ https://issues.apache.org/jira/browse/NIFI-5139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16547087#comment-16547087 ] Otto Fowler commented on NIFI-5139: --- I think we are ok to resolve this then. the reporter can always reopen. > ListenSyslog should process Structured Data > --- > > Key: NIFI-5139 > URL: https://issues.apache.org/jira/browse/NIFI-5139 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.6.0 >Reporter: Aurélien DEHAY >Priority: Minor > > Hello. > > The Structured Data as described in > [https://tools.ietf.org/html/rfc5424#section-6.3] are not read by Nifi. > > I suggest the SD to be put in attributes, prefixed by the ID: > [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"] > to become 3 attributes > * exampleSID@32473-iut = 3 > * exampleSID@32473-eventSource=Application > * exampleSID@32473-eventID=1011 > > This could be useful to preprocess RFC5424 message, prioritize or route them. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5139) ListenSyslog should process Structured Data
[ https://issues.apache.org/jira/browse/NIFI-5139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16547053#comment-16547053 ] Bryan Bende commented on NIFI-5139: --- I think there would be a few options... 1) Lets say you have ListenSyslog with a large batch size, then send to PartitionRecord, and you have a dynamic property like: SID = /structuredData/exampleSID@32473-iut This would partition the records based on the value of that field in the structured data map, then each out going flow file would have an attribute called SID and you could use RouteOnAttribute to route the different SID values to wherever. 2) Another option would to use ListenSyslog or ListenTCPRecord followed by SplitRecord with a record count of 1 and a JsonWriter, then you have one JSON document per flow file and you can use EvaluateJsonPath. 3) You could use ListenSyslog, followed by SplitText with line count of 1, and then ParseSyslog5424 followed by RouteOnAttribute where you can use the attributes like "syslog.structuredData.XYZ" > ListenSyslog should process Structured Data > --- > > Key: NIFI-5139 > URL: https://issues.apache.org/jira/browse/NIFI-5139 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.6.0 >Reporter: Aurélien DEHAY >Priority: Minor > > Hello. > > The Structured Data as described in > [https://tools.ietf.org/html/rfc5424#section-6.3] are not read by Nifi. > > I suggest the SD to be put in attributes, prefixed by the ID: > [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"] > to become 3 attributes > * exampleSID@32473-iut = 3 > * exampleSID@32473-eventSource=Application > * exampleSID@32473-eventID=1011 > > This could be useful to preprocess RFC5424 message, prioritize or route them. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5139) ListenSyslog should process Structured Data
[ https://issues.apache.org/jira/browse/NIFI-5139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16547045#comment-16547045 ] Otto Fowler commented on NIFI-5139: --- Oh, I don't want to make work ;) I don't know off the top of my head how to get the record to attributes, which is the case he wants. But I think that using what we have he can get what he needs in the end. > ListenSyslog should process Structured Data > --- > > Key: NIFI-5139 > URL: https://issues.apache.org/jira/browse/NIFI-5139 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.6.0 >Reporter: Aurélien DEHAY >Priority: Minor > > Hello. > > The Structured Data as described in > [https://tools.ietf.org/html/rfc5424#section-6.3] are not read by Nifi. > > I suggest the SD to be put in attributes, prefixed by the ID: > [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"] > to become 3 attributes > * exampleSID@32473-iut = 3 > * exampleSID@32473-eventSource=Application > * exampleSID@32473-eventID=1011 > > This could be useful to preprocess RFC5424 message, prioritize or route them. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5139) ListenSyslog should process Structured Data
[ https://issues.apache.org/jira/browse/NIFI-5139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16547044#comment-16547044 ] Bryan Bende commented on NIFI-5139: --- Well I already know ListenTCPRecord works with the new syslog readers because I tested that when reviewing your PRs. I was mostly asking to make sure that it solves the intent of this Jira and that I wasn't overlooking something that was being requested. If you want to create an IT test, I don't really see a problem with that, but I'm not trying to create more work here. > ListenSyslog should process Structured Data > --- > > Key: NIFI-5139 > URL: https://issues.apache.org/jira/browse/NIFI-5139 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.6.0 >Reporter: Aurélien DEHAY >Priority: Minor > > Hello. > > The Structured Data as described in > [https://tools.ietf.org/html/rfc5424#section-6.3] are not read by Nifi. > > I suggest the SD to be put in attributes, prefixed by the ID: > [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"] > to become 3 attributes > * exampleSID@32473-iut = 3 > * exampleSID@32473-eventSource=Application > * exampleSID@32473-eventID=1011 > > This could be useful to preprocess RFC5424 message, prioritize or route them. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5139) ListenSyslog should process Structured Data
[ https://issues.apache.org/jira/browse/NIFI-5139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16547034#comment-16547034 ] Otto Fowler commented on NIFI-5139: --- [~bende] how would you feel about new versions of TestListenUDPRecord and TestListenTCPRecord using those readers? I can file the PR under this jira > ListenSyslog should process Structured Data > --- > > Key: NIFI-5139 > URL: https://issues.apache.org/jira/browse/NIFI-5139 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.6.0 >Reporter: Aurélien DEHAY >Priority: Minor > > Hello. > > The Structured Data as described in > [https://tools.ietf.org/html/rfc5424#section-6.3] are not read by Nifi. > > I suggest the SD to be put in attributes, prefixed by the ID: > [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"] > to become 3 attributes > * exampleSID@32473-iut = 3 > * exampleSID@32473-eventSource=Application > * exampleSID@32473-eventID=1011 > > This could be useful to preprocess RFC5424 message, prioritize or route them. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5139) ListenSyslog should process Structured Data
[ https://issues.apache.org/jira/browse/NIFI-5139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16547030#comment-16547030 ] Otto Fowler commented on NIFI-5139: --- Let me check the testing stuff, and see if I can get an IT test or something together for this case. Would that work? > ListenSyslog should process Structured Data > --- > > Key: NIFI-5139 > URL: https://issues.apache.org/jira/browse/NIFI-5139 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.6.0 >Reporter: Aurélien DEHAY >Priority: Minor > > Hello. > > The Structured Data as described in > [https://tools.ietf.org/html/rfc5424#section-6.3] are not read by Nifi. > > I suggest the SD to be put in attributes, prefixed by the ID: > [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"] > to become 3 attributes > * exampleSID@32473-iut = 3 > * exampleSID@32473-eventSource=Application > * exampleSID@32473-eventID=1011 > > This could be useful to preprocess RFC5424 message, prioritize or route them. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5139) ListenSyslog should process Structured Data
[ https://issues.apache.org/jira/browse/NIFI-5139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16547007#comment-16547007 ] Otto Fowler commented on NIFI-5139: --- I think so, but we need to try it. [~adehay] do you think you can take a build from today and use the new Syslog5424RecordReader service? > ListenSyslog should process Structured Data > --- > > Key: NIFI-5139 > URL: https://issues.apache.org/jira/browse/NIFI-5139 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.6.0 >Reporter: Aurélien DEHAY >Priority: Minor > > Hello. > > The Structured Data as described in > [https://tools.ietf.org/html/rfc5424#section-6.3] are not read by Nifi. > > I suggest the SD to be put in attributes, prefixed by the ID: > [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"] > to become 3 attributes > * exampleSID@32473-iut = 3 > * exampleSID@32473-eventSource=Application > * exampleSID@32473-eventID=1011 > > This could be useful to preprocess RFC5424 message, prioritize or route them. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (NIFI-5324) Implement syslog record readers
[ https://issues.apache.org/jira/browse/NIFI-5324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Otto Fowler resolved NIFI-5324. --- Resolution: Implemented > Implement syslog record readers > --- > > Key: NIFI-5324 > URL: https://issues.apache.org/jira/browse/NIFI-5324 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Bryan Bende >Assignee: Otto Fowler >Priority: Major > > Creating this Jira based on discussion with [~ottobackwards] in the NiFi > HipChat room... > We currently have ListenSyslog with optional parsing when batch size is 1, > and ParseSyslog which also assumes 1 message per flow file. There is also > ListenTCPRecord and ListenUDPRecord which can be used with a GrokReader to > read log messages from the respective network connections. > The common scenario for wanting to parse the syslog messages is to extract a > field from the syslog message into an attribute and then use the attribute to > make decisions like routing/filtering. > Since the "1 message per flow file" pattern is generally something we try to > avoid, it would be nice if we could keep batches of syslog messages together > in a single flow file and then use record processors to process the batches. > For example, if we had a syslog record reader we could then use > PartitionRecord to divide a flow file of many syslog records into smaller > groups based on some field in the message, each group can then be routed > somewhere based on the group value. > Another example would be to use QueryRecord to run a SQL query that selects > specify syslog messages based on a field in the message. > It would also make it easy to convert syslog messages to a structured format > using ConvertRecord with a syslog reader and a writer like JSON or Avro. > We would likely want two syslog record readers, one for each of the RFC > formats. > One aspect to consider is related to the schema used/produced by the > reader... typically the readers/writers have a "Schema Access Strategy" where > they can obtain a schema from a schema registry, or from flow file > attributes, or something specific to the format like an embedded Avro schema. > In this case, the schema is somewhat pre-determined by the specific syslog > reader because the schema can only be at-most the fields produced by the > reader parsing the messages. So this may be a case where there is no schema > access strategy, and there are per-determined schemas. It is sort of like > the GrokReader where it creates a schema from the named fields in the > expression, except in this case there is no user defined expression, and the > named fields are dictated by the parser. > We may need to reuse syslog related code that is in nifi-standard-processors, > so it might require moving that code to nifi-processor-utils, or creating a > new nifi-syslog-utils module. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5436) TestRunner should allow setting the processor name
[ https://issues.apache.org/jira/browse/NIFI-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16547002#comment-16547002 ] ASF GitHub Bot commented on NIFI-5436: -- Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/nifi/pull/2902#discussion_r203146804 --- Diff: nifi-docs/src/main/asciidoc/developer-guide.adoc --- @@ -2144,9 +2144,10 @@ you can use the Maven dependency: -We create a new `TestRunner` by calling the static `newTestRunner` method of the `TestRunners` class -(located in the `org.apache.nifi.util` package). This method takes a single argument. That argument can -either be the class of the Processor to test or can be an instance of a Processor. +We create a new `TestRunner` by calling one of the static `newTestRunner` methods of the `TestRunners` class +(located in the `org.apache.nifi.util` package). These methods take an argument can --- End diff -- change going in now, please check @alopresto > TestRunner should allow setting the processor name > -- > > Key: NIFI-5436 > URL: https://issues.apache.org/jira/browse/NIFI-5436 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > > Currently, the ProcessContext.getName() just returns a blank String. Some > users use that value of the context, but cannot test their processors > properly because there is no way to set this value. > > The TestRunner down to the MockProcessContext should support setting the name > of the processor when you create the TestRunner instance. > > If no name is provided, the the Processor class name should be used. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #2902: NIFI-5436 add ability to set the processor name in ...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/nifi/pull/2902#discussion_r203146804 --- Diff: nifi-docs/src/main/asciidoc/developer-guide.adoc --- @@ -2144,9 +2144,10 @@ you can use the Maven dependency: -We create a new `TestRunner` by calling the static `newTestRunner` method of the `TestRunners` class -(located in the `org.apache.nifi.util` package). This method takes a single argument. That argument can -either be the class of the Processor to test or can be an instance of a Processor. +We create a new `TestRunner` by calling one of the static `newTestRunner` methods of the `TestRunners` class +(located in the `org.apache.nifi.util` package). These methods take an argument can --- End diff -- change going in now, please check @alopresto ---
[jira] [Commented] (NIFI-4216) NiFi Maven plugin for Nars needs to be marked ThreadSafe
[ https://issues.apache.org/jira/browse/NIFI-4216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546984#comment-16546984 ] Bryan Bende commented on NIFI-4216: --- [~joewitt] I tried a parallel build of NiFi using the NAR Plugin with this patch, and it appears to build correctly without the warnings. Let me know if you wanted to do further review, otherwise happy to merge. > NiFi Maven plugin for Nars needs to be marked ThreadSafe > > > Key: NIFI-4216 > URL: https://issues.apache.org/jira/browse/NIFI-4216 > Project: Apache NiFi > Issue Type: Bug > Components: Tools and Build >Reporter: Joseph Witt >Assignee: Joseph Witt >Priority: Major > > Building in Maven 3.5.0 reveals the following warning (all works fine but...) > [WARNING] * > [WARNING] * Your build is requesting parallel execution, but project * > [WARNING] * contains the following plugin(s) that have goals not marked * > [WARNING] * as @threadSafe to support parallel building. * > [WARNING] * While this /may/ work fine, please look for plugin updates* > [WARNING] * and/or request plugins be made thread-safe. * > [WARNING] * If reporting an issue, report it against the plugin in* > [WARNING] * question, not against maven-core * > [WARNING] * > [WARNING] The following plugins are not marked @threadSafe in > nifi-framework-nar: > [WARNING] org.apache.nifi:nifi-nar-maven-plugin:1.2.0 > [WARNING] Enable debug to see more precisely which goals are not marked > @threadSafe. > [WARNING] * -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-4930) Nar-Dependency-Version - timestamped snapshot version problem
[ https://issues.apache.org/jira/browse/NIFI-4930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Bende updated NIFI-4930: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Nar-Dependency-Version - timestamped snapshot version problem > - > > Key: NIFI-4930 > URL: https://issues.apache.org/jira/browse/NIFI-4930 > Project: Apache NiFi > Issue Type: Bug > Components: Tools and Build >Affects Versions: 1.2.0 >Reporter: Arne Degenring >Assignee: Bryan Bende >Priority: Major > Labels: Nar-Maven-Plugin > Fix For: nifi-nar-maven-plugin-1.2.1 > > Original Estimate: 5m > Remaining Estimate: 5m > > The NAR Maven Plugin 1.2.0 is using artifact.getVersion() to determine the > Nar-Dependency-Version, however this version number can be a timestamped > snapshot version number if the dependency was downloaded from a remote repo. > The Nar-Dependency-Version can then differ from the declared Nar-Version in > the other JAR. > Suggested Fix in NarMojo.java: use artifact.getBaseVersion() instead of > artifact.getVersion(): > Replace line 705: > narDependency = new NarDependency(artifact.getGroupId(), > artifact.getArtifactId(), artifact.getVersion()); > with: > narDependency = new NarDependency(artifact.getGroupId(), > artifact.getArtifactId(), artifact.getBaseVersion()); > > -- > Long explanation, from > [https://lists.apache.org/thread.html/ce44faa7b1a54aac417fca06200712de629c3bc2ed692813c0a10a73@%3Cusers.nifi.apache.org%3E] > We have seen a strange problem on NiFi 1.4.0 where custom processors could > suddenly not be started, because of incompatibility with custom services: > 2018-03-02 13:40:35,490 ERROR [main] o.apache.nifi.controller.FlowController > Unable to start ...[id=5d57d39a-015c-1000--d654d90b] due to > java.lang.IllegalStateException: Processor ... is not in a valid state due to > ... is invalid because ... - 1.4-SNAPSHOT from ... is not compatible with > ... - 1.4-SNAPSHOT ...] > It seems that the root cause was related to: > 2018-03-02 13:39:55,086 WARN [main] org.apache.nifi.nar.NarClassLoaders While > loading ...:...:1.5-SNAPSHOT' unable to locate exact NAR dependency > '...:...:1.4-20180302.33-16'. Only found one possible match > ...::1.4-SNAPSHOT'. Continuing... > It turned out that our various custom NARs were built on different built > agents. > Maven has the (ugly) behaviour that when a snapshot version is present in the > local repository after local build, the version number will be e.g. > 1.4-SNAPSHOT. However if it is downloaded from a remote repository, the > version number includes a timestamp such as 1.4-20180302.33-16. > So the manifest of the depending NARs contained: > Nar-Dependency-Version: 1.4-20180302.33-16 > while the other NAR file declared: > Nar-Version: 1.4-SNAPSHOT > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4930) Nar-Dependency-Version - timestamped snapshot version problem
[ https://issues.apache.org/jira/browse/NIFI-4930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546972#comment-16546972 ] ASF GitHub Bot commented on NIFI-4930: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi-maven/pull/3 > Nar-Dependency-Version - timestamped snapshot version problem > - > > Key: NIFI-4930 > URL: https://issues.apache.org/jira/browse/NIFI-4930 > Project: Apache NiFi > Issue Type: Bug > Components: Tools and Build >Affects Versions: 1.2.0 >Reporter: Arne Degenring >Assignee: Bryan Bende >Priority: Major > Labels: Nar-Maven-Plugin > Fix For: nifi-nar-maven-plugin-1.2.1 > > Original Estimate: 5m > Remaining Estimate: 5m > > The NAR Maven Plugin 1.2.0 is using artifact.getVersion() to determine the > Nar-Dependency-Version, however this version number can be a timestamped > snapshot version number if the dependency was downloaded from a remote repo. > The Nar-Dependency-Version can then differ from the declared Nar-Version in > the other JAR. > Suggested Fix in NarMojo.java: use artifact.getBaseVersion() instead of > artifact.getVersion(): > Replace line 705: > narDependency = new NarDependency(artifact.getGroupId(), > artifact.getArtifactId(), artifact.getVersion()); > with: > narDependency = new NarDependency(artifact.getGroupId(), > artifact.getArtifactId(), artifact.getBaseVersion()); > > -- > Long explanation, from > [https://lists.apache.org/thread.html/ce44faa7b1a54aac417fca06200712de629c3bc2ed692813c0a10a73@%3Cusers.nifi.apache.org%3E] > We have seen a strange problem on NiFi 1.4.0 where custom processors could > suddenly not be started, because of incompatibility with custom services: > 2018-03-02 13:40:35,490 ERROR [main] o.apache.nifi.controller.FlowController > Unable to start ...[id=5d57d39a-015c-1000--d654d90b] due to > java.lang.IllegalStateException: Processor ... is not in a valid state due to > ... is invalid because ... - 1.4-SNAPSHOT from ... is not compatible with > ... - 1.4-SNAPSHOT ...] > It seems that the root cause was related to: > 2018-03-02 13:39:55,086 WARN [main] org.apache.nifi.nar.NarClassLoaders While > loading ...:...:1.5-SNAPSHOT' unable to locate exact NAR dependency > '...:...:1.4-20180302.33-16'. Only found one possible match > ...::1.4-SNAPSHOT'. Continuing... > It turned out that our various custom NARs were built on different built > agents. > Maven has the (ugly) behaviour that when a snapshot version is present in the > local repository after local build, the version number will be e.g. > 1.4-SNAPSHOT. However if it is downloaded from a remote repository, the > version number includes a timestamp such as 1.4-20180302.33-16. > So the manifest of the depending NARs contained: > Nar-Dependency-Version: 1.4-20180302.33-16 > while the other NAR file declared: > Nar-Version: 1.4-SNAPSHOT > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4930) Nar-Dependency-Version - timestamped snapshot version problem
[ https://issues.apache.org/jira/browse/NIFI-4930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546971#comment-16546971 ] Bryan Bende commented on NIFI-4930: --- Sorry for the delay on trying this, but it looks good to me, going to merge, thanks! > Nar-Dependency-Version - timestamped snapshot version problem > - > > Key: NIFI-4930 > URL: https://issues.apache.org/jira/browse/NIFI-4930 > Project: Apache NiFi > Issue Type: Bug > Components: Tools and Build >Affects Versions: 1.2.0 >Reporter: Arne Degenring >Assignee: Bryan Bende >Priority: Major > Labels: Nar-Maven-Plugin > Fix For: nifi-nar-maven-plugin-1.2.1 > > Original Estimate: 5m > Remaining Estimate: 5m > > The NAR Maven Plugin 1.2.0 is using artifact.getVersion() to determine the > Nar-Dependency-Version, however this version number can be a timestamped > snapshot version number if the dependency was downloaded from a remote repo. > The Nar-Dependency-Version can then differ from the declared Nar-Version in > the other JAR. > Suggested Fix in NarMojo.java: use artifact.getBaseVersion() instead of > artifact.getVersion(): > Replace line 705: > narDependency = new NarDependency(artifact.getGroupId(), > artifact.getArtifactId(), artifact.getVersion()); > with: > narDependency = new NarDependency(artifact.getGroupId(), > artifact.getArtifactId(), artifact.getBaseVersion()); > > -- > Long explanation, from > [https://lists.apache.org/thread.html/ce44faa7b1a54aac417fca06200712de629c3bc2ed692813c0a10a73@%3Cusers.nifi.apache.org%3E] > We have seen a strange problem on NiFi 1.4.0 where custom processors could > suddenly not be started, because of incompatibility with custom services: > 2018-03-02 13:40:35,490 ERROR [main] o.apache.nifi.controller.FlowController > Unable to start ...[id=5d57d39a-015c-1000--d654d90b] due to > java.lang.IllegalStateException: Processor ... is not in a valid state due to > ... is invalid because ... - 1.4-SNAPSHOT from ... is not compatible with > ... - 1.4-SNAPSHOT ...] > It seems that the root cause was related to: > 2018-03-02 13:39:55,086 WARN [main] org.apache.nifi.nar.NarClassLoaders While > loading ...:...:1.5-SNAPSHOT' unable to locate exact NAR dependency > '...:...:1.4-20180302.33-16'. Only found one possible match > ...::1.4-SNAPSHOT'. Continuing... > It turned out that our various custom NARs were built on different built > agents. > Maven has the (ugly) behaviour that when a snapshot version is present in the > local repository after local build, the version number will be e.g. > 1.4-SNAPSHOT. However if it is downloaded from a remote repository, the > version number includes a timestamp such as 1.4-20180302.33-16. > So the manifest of the depending NARs contained: > Nar-Dependency-Version: 1.4-20180302.33-16 > while the other NAR file declared: > Nar-Version: 1.4-SNAPSHOT > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-maven pull request #3: NIFI-4930: Resolve issues when fetching nar file...
Github user asfgit closed the pull request at: https://github.com/apache/nifi-maven/pull/3 ---
[jira] [Updated] (NIFI-4930) Nar-Dependency-Version - timestamped snapshot version problem
[ https://issues.apache.org/jira/browse/NIFI-4930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Bende updated NIFI-4930: -- Fix Version/s: nifi-nar-maven-plugin-1.2.1 > Nar-Dependency-Version - timestamped snapshot version problem > - > > Key: NIFI-4930 > URL: https://issues.apache.org/jira/browse/NIFI-4930 > Project: Apache NiFi > Issue Type: Bug > Components: Tools and Build >Affects Versions: 1.2.0 >Reporter: Arne Degenring >Assignee: Bryan Bende >Priority: Major > Labels: Nar-Maven-Plugin > Fix For: nifi-nar-maven-plugin-1.2.1 > > Original Estimate: 5m > Remaining Estimate: 5m > > The NAR Maven Plugin 1.2.0 is using artifact.getVersion() to determine the > Nar-Dependency-Version, however this version number can be a timestamped > snapshot version number if the dependency was downloaded from a remote repo. > The Nar-Dependency-Version can then differ from the declared Nar-Version in > the other JAR. > Suggested Fix in NarMojo.java: use artifact.getBaseVersion() instead of > artifact.getVersion(): > Replace line 705: > narDependency = new NarDependency(artifact.getGroupId(), > artifact.getArtifactId(), artifact.getVersion()); > with: > narDependency = new NarDependency(artifact.getGroupId(), > artifact.getArtifactId(), artifact.getBaseVersion()); > > -- > Long explanation, from > [https://lists.apache.org/thread.html/ce44faa7b1a54aac417fca06200712de629c3bc2ed692813c0a10a73@%3Cusers.nifi.apache.org%3E] > We have seen a strange problem on NiFi 1.4.0 where custom processors could > suddenly not be started, because of incompatibility with custom services: > 2018-03-02 13:40:35,490 ERROR [main] o.apache.nifi.controller.FlowController > Unable to start ...[id=5d57d39a-015c-1000--d654d90b] due to > java.lang.IllegalStateException: Processor ... is not in a valid state due to > ... is invalid because ... - 1.4-SNAPSHOT from ... is not compatible with > ... - 1.4-SNAPSHOT ...] > It seems that the root cause was related to: > 2018-03-02 13:39:55,086 WARN [main] org.apache.nifi.nar.NarClassLoaders While > loading ...:...:1.5-SNAPSHOT' unable to locate exact NAR dependency > '...:...:1.4-20180302.33-16'. Only found one possible match > ...::1.4-SNAPSHOT'. Continuing... > It turned out that our various custom NARs were built on different built > agents. > Maven has the (ugly) behaviour that when a snapshot version is present in the > local repository after local build, the version number will be e.g. > 1.4-SNAPSHOT. However if it is downloaded from a remote repository, the > version number includes a timestamp such as 1.4-20180302.33-16. > So the manifest of the depending NARs contained: > Nar-Dependency-Version: 1.4-20180302.33-16 > while the other NAR file declared: > Nar-Version: 1.4-SNAPSHOT > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (NIFI-4930) Nar-Dependency-Version - timestamped snapshot version problem
[ https://issues.apache.org/jira/browse/NIFI-4930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Bende reassigned NIFI-4930: - Assignee: Bryan Bende (was: Bryan Bende) > Nar-Dependency-Version - timestamped snapshot version problem > - > > Key: NIFI-4930 > URL: https://issues.apache.org/jira/browse/NIFI-4930 > Project: Apache NiFi > Issue Type: Bug > Components: Tools and Build >Affects Versions: 1.2.0 >Reporter: Arne Degenring >Assignee: Bryan Bende >Priority: Major > Labels: Nar-Maven-Plugin > Original Estimate: 5m > Remaining Estimate: 5m > > The NAR Maven Plugin 1.2.0 is using artifact.getVersion() to determine the > Nar-Dependency-Version, however this version number can be a timestamped > snapshot version number if the dependency was downloaded from a remote repo. > The Nar-Dependency-Version can then differ from the declared Nar-Version in > the other JAR. > Suggested Fix in NarMojo.java: use artifact.getBaseVersion() instead of > artifact.getVersion(): > Replace line 705: > narDependency = new NarDependency(artifact.getGroupId(), > artifact.getArtifactId(), artifact.getVersion()); > with: > narDependency = new NarDependency(artifact.getGroupId(), > artifact.getArtifactId(), artifact.getBaseVersion()); > > -- > Long explanation, from > [https://lists.apache.org/thread.html/ce44faa7b1a54aac417fca06200712de629c3bc2ed692813c0a10a73@%3Cusers.nifi.apache.org%3E] > We have seen a strange problem on NiFi 1.4.0 where custom processors could > suddenly not be started, because of incompatibility with custom services: > 2018-03-02 13:40:35,490 ERROR [main] o.apache.nifi.controller.FlowController > Unable to start ...[id=5d57d39a-015c-1000--d654d90b] due to > java.lang.IllegalStateException: Processor ... is not in a valid state due to > ... is invalid because ... - 1.4-SNAPSHOT from ... is not compatible with > ... - 1.4-SNAPSHOT ...] > It seems that the root cause was related to: > 2018-03-02 13:39:55,086 WARN [main] org.apache.nifi.nar.NarClassLoaders While > loading ...:...:1.5-SNAPSHOT' unable to locate exact NAR dependency > '...:...:1.4-20180302.33-16'. Only found one possible match > ...::1.4-SNAPSHOT'. Continuing... > It turned out that our various custom NARs were built on different built > agents. > Maven has the (ugly) behaviour that when a snapshot version is present in the > local repository after local build, the version number will be e.g. > 1.4-SNAPSHOT. However if it is downloaded from a remote repository, the > version number includes a timestamp such as 1.4-20180302.33-16. > So the manifest of the depending NARs contained: > Nar-Dependency-Version: 1.4-20180302.33-16 > while the other NAR file declared: > Nar-Version: 1.4-SNAPSHOT > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (NIFI-5406) Add new listing strategy by tracking listed entities to ListXXXX processors
[ https://issues.apache.org/jira/browse/NIFI-5406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne resolved NIFI-5406. -- Resolution: Fixed Fix Version/s: 1.8.0 > Add new listing strategy by tracking listed entities to List processors > --- > > Key: NIFI-5406 > URL: https://issues.apache.org/jira/browse/NIFI-5406 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Koji Kawamura >Assignee: Koji Kawamura >Priority: Major > Fix For: 1.8.0 > > > Current List processors (ListFile, ListFTP, ListSFTP ... etc) implementation > relies on file last modified timestamp to pick new or updated files. This > approach is efficient and lightweight in terms of state management, because > it only tracks latest modified timestamp and last executed timestamp. > However, timestamps do not work as expected in some file systems, causing > List processors missing files periodically. See NIFI-3332 comments for > details. > In order to pick every entity that has not seen before or has been updated > since it had seen last time, we need another set of processors using > different approach, that is by tracking listed entities: > * Add new abstract processor AbstractWatchEntries similar to > AbstractListProcessor but uses different approach > * Target entities have: name (path), size and last-modified-timestamp > * Implementation Processors have following properties: > ** 'Watch Time Window' to limit the maximum time period to hold the already > listed entries. E.g. if set as '30min', the processor keeps entities listed > in the last 30 mins. > ** 'Minimum File Age' to defer listing entities potentially being written > * Any entity added but not listed ever having last-modified-timestamp older > than configured 'Watch Time Window' will not be listed. If user needs to pick > these items, they have to make 'Watch Time Window' longer. It also increases > the size of data the processor has to persist in the K/V store. Efficiency vs > reliability trade-off. > * The already-listed entities are persisted into one of supported K/V store > through DistributedMapCacheClient service. User can chose what KVS to use > from HBase, Redis, Couchbase and File (DistributedMapCacheServer with > persistence file). > * The reason to use KVS instead of ManagedState is, to avoid hammering > Zookeeper too much with frequently updating Zk node with large amount of > data. The number of already-listed entries can be huge depending on > use-cases. Also, we can compress entities with DistributedMapCacheClient as > it supports putting byte array, while ManagedState only supports Map String>. > * On each onTrigger: > ** Processor performs listing. Listed entries meeting any of the following > condition will be written to the 'success' output FlowFile: > *** Not exists in the already-listed entities > *** Having newer last-modified-timestamp > *** Having different size > ** Already listed entries those are old enough compared to 'Watch Time > Window' are discarded from the already-listed entries. > * Initial supporting target is Local file system, FTP and SFTP -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5406) Add new listing strategy by tracking listed entities to ListXXXX processors
[ https://issues.apache.org/jira/browse/NIFI-5406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546954#comment-16546954 ] ASF GitHub Bot commented on NIFI-5406: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/2876 > Add new listing strategy by tracking listed entities to List processors > --- > > Key: NIFI-5406 > URL: https://issues.apache.org/jira/browse/NIFI-5406 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Koji Kawamura >Assignee: Koji Kawamura >Priority: Major > Fix For: 1.8.0 > > > Current List processors (ListFile, ListFTP, ListSFTP ... etc) implementation > relies on file last modified timestamp to pick new or updated files. This > approach is efficient and lightweight in terms of state management, because > it only tracks latest modified timestamp and last executed timestamp. > However, timestamps do not work as expected in some file systems, causing > List processors missing files periodically. See NIFI-3332 comments for > details. > In order to pick every entity that has not seen before or has been updated > since it had seen last time, we need another set of processors using > different approach, that is by tracking listed entities: > * Add new abstract processor AbstractWatchEntries similar to > AbstractListProcessor but uses different approach > * Target entities have: name (path), size and last-modified-timestamp > * Implementation Processors have following properties: > ** 'Watch Time Window' to limit the maximum time period to hold the already > listed entries. E.g. if set as '30min', the processor keeps entities listed > in the last 30 mins. > ** 'Minimum File Age' to defer listing entities potentially being written > * Any entity added but not listed ever having last-modified-timestamp older > than configured 'Watch Time Window' will not be listed. If user needs to pick > these items, they have to make 'Watch Time Window' longer. It also increases > the size of data the processor has to persist in the K/V store. Efficiency vs > reliability trade-off. > * The already-listed entities are persisted into one of supported K/V store > through DistributedMapCacheClient service. User can chose what KVS to use > from HBase, Redis, Couchbase and File (DistributedMapCacheServer with > persistence file). > * The reason to use KVS instead of ManagedState is, to avoid hammering > Zookeeper too much with frequently updating Zk node with large amount of > data. The number of already-listed entries can be huge depending on > use-cases. Also, we can compress entities with DistributedMapCacheClient as > it supports putting byte array, while ManagedState only supports Map String>. > * On each onTrigger: > ** Processor performs listing. Listed entries meeting any of the following > condition will be written to the 'success' output FlowFile: > *** Not exists in the already-listed entities > *** Having newer last-modified-timestamp > *** Having different size > ** Already listed entries those are old enough compared to 'Watch Time > Window' are discarded from the already-listed entries. > * Initial supporting target is Local file system, FTP and SFTP -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #2876: NIFI-5406: Adding new listing strategy to List proc...
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/2876 ---
[jira] [Commented] (NIFI-5406) Add new listing strategy by tracking listed entities to ListXXXX processors
[ https://issues.apache.org/jira/browse/NIFI-5406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546953#comment-16546953 ] ASF GitHub Bot commented on NIFI-5406: -- Github user markap14 commented on the issue: https://github.com/apache/nifi/pull/2876 All of my testing looks good as well. +1 merged to master. Thanks for the improvement, @ijokarumawak ! > Add new listing strategy by tracking listed entities to List processors > --- > > Key: NIFI-5406 > URL: https://issues.apache.org/jira/browse/NIFI-5406 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Koji Kawamura >Assignee: Koji Kawamura >Priority: Major > > Current List processors (ListFile, ListFTP, ListSFTP ... etc) implementation > relies on file last modified timestamp to pick new or updated files. This > approach is efficient and lightweight in terms of state management, because > it only tracks latest modified timestamp and last executed timestamp. > However, timestamps do not work as expected in some file systems, causing > List processors missing files periodically. See NIFI-3332 comments for > details. > In order to pick every entity that has not seen before or has been updated > since it had seen last time, we need another set of processors using > different approach, that is by tracking listed entities: > * Add new abstract processor AbstractWatchEntries similar to > AbstractListProcessor but uses different approach > * Target entities have: name (path), size and last-modified-timestamp > * Implementation Processors have following properties: > ** 'Watch Time Window' to limit the maximum time period to hold the already > listed entries. E.g. if set as '30min', the processor keeps entities listed > in the last 30 mins. > ** 'Minimum File Age' to defer listing entities potentially being written > * Any entity added but not listed ever having last-modified-timestamp older > than configured 'Watch Time Window' will not be listed. If user needs to pick > these items, they have to make 'Watch Time Window' longer. It also increases > the size of data the processor has to persist in the K/V store. Efficiency vs > reliability trade-off. > * The already-listed entities are persisted into one of supported K/V store > through DistributedMapCacheClient service. User can chose what KVS to use > from HBase, Redis, Couchbase and File (DistributedMapCacheServer with > persistence file). > * The reason to use KVS instead of ManagedState is, to avoid hammering > Zookeeper too much with frequently updating Zk node with large amount of > data. The number of already-listed entries can be huge depending on > use-cases. Also, we can compress entities with DistributedMapCacheClient as > it supports putting byte array, while ManagedState only supports Map String>. > * On each onTrigger: > ** Processor performs listing. Listed entries meeting any of the following > condition will be written to the 'success' output FlowFile: > *** Not exists in the already-listed entities > *** Having newer last-modified-timestamp > *** Having different size > ** Already listed entries those are old enough compared to 'Watch Time > Window' are discarded from the already-listed entries. > * Initial supporting target is Local file system, FTP and SFTP -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2876: NIFI-5406: Adding new listing strategy to List processors.
Github user markap14 commented on the issue: https://github.com/apache/nifi/pull/2876 All of my testing looks good as well. +1 merged to master. Thanks for the improvement, @ijokarumawak ! ---
[jira] [Commented] (NIFI-5436) TestRunner should allow setting the processor name
[ https://issues.apache.org/jira/browse/NIFI-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546942#comment-16546942 ] ASF GitHub Bot commented on NIFI-5436: -- Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/2902 Ran `contrib-check` and all tests pass. If you make the additional change in the documentation, I am a +1. > TestRunner should allow setting the processor name > -- > > Key: NIFI-5436 > URL: https://issues.apache.org/jira/browse/NIFI-5436 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > > Currently, the ProcessContext.getName() just returns a blank String. Some > users use that value of the context, but cannot test their processors > properly because there is no way to set this value. > > The TestRunner down to the MockProcessContext should support setting the name > of the processor when you create the TestRunner instance. > > If no name is provided, the the Processor class name should be used. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2902: NIFI-5436 add ability to set the processor name in the Pro...
Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/2902 Ran `contrib-check` and all tests pass. If you make the additional change in the documentation, I am a +1. ---
[jira] [Commented] (NIFI-5436) TestRunner should allow setting the processor name
[ https://issues.apache.org/jira/browse/NIFI-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546937#comment-16546937 ] ASF GitHub Bot commented on NIFI-5436: -- Github user alopresto commented on a diff in the pull request: https://github.com/apache/nifi/pull/2902#discussion_r203117425 --- Diff: nifi-docs/src/main/asciidoc/developer-guide.adoc --- @@ -2144,9 +2144,10 @@ you can use the Maven dependency: -We create a new `TestRunner` by calling the static `newTestRunner` method of the `TestRunners` class -(located in the `org.apache.nifi.util` package). This method takes a single argument. That argument can -either be the class of the Processor to test or can be an instance of a Processor. +We create a new `TestRunner` by calling one of the static `newTestRunner` methods of the `TestRunners` class +(located in the `org.apache.nifi.util` package). These methods take an argument can --- End diff -- Sorry, missed this construct: "These methods take an argument (can either be the class of the Processor to test or can be an instance of a Processor), and allow the setting of the processor name as well." (add parentheses or commas to separate the relative clause). > TestRunner should allow setting the processor name > -- > > Key: NIFI-5436 > URL: https://issues.apache.org/jira/browse/NIFI-5436 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > > Currently, the ProcessContext.getName() just returns a blank String. Some > users use that value of the context, but cannot test their processors > properly because there is no way to set this value. > > The TestRunner down to the MockProcessContext should support setting the name > of the processor when you create the TestRunner instance. > > If no name is provided, the the Processor class name should be used. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #2902: NIFI-5436 add ability to set the processor name in ...
Github user alopresto commented on a diff in the pull request: https://github.com/apache/nifi/pull/2902#discussion_r203117425 --- Diff: nifi-docs/src/main/asciidoc/developer-guide.adoc --- @@ -2144,9 +2144,10 @@ you can use the Maven dependency: -We create a new `TestRunner` by calling the static `newTestRunner` method of the `TestRunners` class -(located in the `org.apache.nifi.util` package). This method takes a single argument. That argument can -either be the class of the Processor to test or can be an instance of a Processor. +We create a new `TestRunner` by calling one of the static `newTestRunner` methods of the `TestRunners` class +(located in the `org.apache.nifi.util` package). These methods take an argument can --- End diff -- Sorry, missed this construct: "These methods take an argument (can either be the class of the Processor to test or can be an instance of a Processor), and allow the setting of the processor name as well." (add parentheses or commas to separate the relative clause). ---
[jira] [Commented] (MINIFICPP-556) Create snap build
[ https://issues.apache.org/jira/browse/MINIFICPP-556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546933#comment-16546933 ] ASF GitHub Bot commented on MINIFICPP-556: -- Github user phrocker commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/371 @jdye64 I was able to with some documentation you provided offline; however, following guides available online previously it failed with the error I sent you. Perhaps it would be useful to add something to the documentation in addition to this snapcraft file? > Create snap build > - > > Key: MINIFICPP-556 > URL: https://issues.apache.org/jira/browse/MINIFICPP-556 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Andrew Christianson >Assignee: Andrew Christianson >Priority: Major > > Package nifi-minifi-cpp into a snap package for easy distribution to users > who use snap. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp issue #371: MINIFICPP-556 Added initial snapcraft build
Github user phrocker commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/371 @jdye64 I was able to with some documentation you provided offline; however, following guides available online previously it failed with the error I sent you. Perhaps it would be useful to add something to the documentation in addition to this snapcraft file? ---
[jira] [Created] (MINIFICPP-560) Create deb package target in cmake build
Andrew Christianson created MINIFICPP-560: - Summary: Create deb package target in cmake build Key: MINIFICPP-560 URL: https://issues.apache.org/jira/browse/MINIFICPP-560 Project: NiFi MiNiFi C++ Issue Type: Improvement Reporter: Andrew Christianson Assignee: Andrew Christianson Support packaging to deb for easier install and ability to have minificpp in repositories. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5139) ListenSyslog should process Structured Data
[ https://issues.apache.org/jira/browse/NIFI-5139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546929#comment-16546929 ] Bryan Bende commented on NIFI-5139: --- [~adehay] [~ottobackwards] I think this can now be accomplished by using ListenTCPRecord/ListenUDPRecord with the Syslog5424 record reader, what do you guys think? Maybe we can close this one. > ListenSyslog should process Structured Data > --- > > Key: NIFI-5139 > URL: https://issues.apache.org/jira/browse/NIFI-5139 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.6.0 >Reporter: Aurélien DEHAY >Priority: Minor > > Hello. > > The Structured Data as described in > [https://tools.ietf.org/html/rfc5424#section-6.3] are not read by Nifi. > > I suggest the SD to be put in attributes, prefixed by the ID: > [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"] > to become 3 attributes > * exampleSID@32473-iut = 3 > * exampleSID@32473-eventSource=Application > * exampleSID@32473-eventID=1011 > > This could be useful to preprocess RFC5424 message, prioritize or route them. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MINIFICPP-556) Create snap build
[ https://issues.apache.org/jira/browse/MINIFICPP-556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546928#comment-16546928 ] ASF GitHub Bot commented on MINIFICPP-556: -- Github user jdye64 commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/371 @phrocker were you able to resolve your issues? I didn't experience any problems during my testing. > Create snap build > - > > Key: MINIFICPP-556 > URL: https://issues.apache.org/jira/browse/MINIFICPP-556 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Andrew Christianson >Assignee: Andrew Christianson >Priority: Major > > Package nifi-minifi-cpp into a snap package for easy distribution to users > who use snap. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp issue #371: MINIFICPP-556 Added initial snapcraft build
Github user jdye64 commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/371 @phrocker were you able to resolve your issues? I didn't experience any problems during my testing. ---
[jira] [Resolved] (NIFI-5432) Add Syslog Record Reader legacy Syslog
[ https://issues.apache.org/jira/browse/NIFI-5432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Bende resolved NIFI-5432. --- Resolution: Fixed Fix Version/s: 1.8.0 > Add Syslog Record Reader legacy Syslog > -- > > Key: NIFI-5432 > URL: https://issues.apache.org/jira/browse/NIFI-5432 > Project: Apache NiFi > Issue Type: Sub-task >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > Fix For: 1.8.0 > > > Create a record reader, based on nifi-syslog-utils for reading syslog without > support for structured data -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5432) Add Syslog Record Reader legacy Syslog
[ https://issues.apache.org/jira/browse/NIFI-5432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546923#comment-16546923 ] ASF GitHub Bot commented on NIFI-5432: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/2900 > Add Syslog Record Reader legacy Syslog > -- > > Key: NIFI-5432 > URL: https://issues.apache.org/jira/browse/NIFI-5432 > Project: Apache NiFi > Issue Type: Sub-task >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > > Create a record reader, based on nifi-syslog-utils for reading syslog without > support for structured data -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5432) Add Syslog Record Reader legacy Syslog
[ https://issues.apache.org/jira/browse/NIFI-5432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546922#comment-16546922 ] ASF subversion and git services commented on NIFI-5432: --- Commit 0a493bf7fdac5b16225e6e692d2a03e095765f7a in nifi's branch refs/heads/master from [~ottobackwards] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=0a493bf ] NIFI-5432 Add Syslog Record Reader legacy Syslog - Add additional details for schema This closes #2900. Signed-off-by: Bryan Bende > Add Syslog Record Reader legacy Syslog > -- > > Key: NIFI-5432 > URL: https://issues.apache.org/jira/browse/NIFI-5432 > Project: Apache NiFi > Issue Type: Sub-task >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > > Create a record reader, based on nifi-syslog-utils for reading syslog without > support for structured data -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #2900: NIFI-5432 Add Syslog Record Reader legacy Syslog
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/2900 ---
[jira] [Commented] (NIFI-5432) Add Syslog Record Reader legacy Syslog
[ https://issues.apache.org/jira/browse/NIFI-5432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546921#comment-16546921 ] ASF GitHub Bot commented on NIFI-5432: -- Github user bbende commented on the issue: https://github.com/apache/nifi/pull/2900 Looks good, merging, thanks! > Add Syslog Record Reader legacy Syslog > -- > > Key: NIFI-5432 > URL: https://issues.apache.org/jira/browse/NIFI-5432 > Project: Apache NiFi > Issue Type: Sub-task >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > > Create a record reader, based on nifi-syslog-utils for reading syslog without > support for structured data -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2900: NIFI-5432 Add Syslog Record Reader legacy Syslog
Github user bbende commented on the issue: https://github.com/apache/nifi/pull/2900 Looks good, merging, thanks! ---
[jira] [Commented] (NIFI-5406) Add new listing strategy by tracking listed entities to ListXXXX processors
[ https://issues.apache.org/jira/browse/NIFI-5406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546911#comment-16546911 ] ASF GitHub Bot commented on NIFI-5406: -- Github user markap14 commented on the issue: https://github.com/apache/nifi/pull/2876 @ijokarumawak thanks for the updates! Code looks good! I will run a few tests and if all looks good will merge to master. Thanks! > Add new listing strategy by tracking listed entities to List processors > --- > > Key: NIFI-5406 > URL: https://issues.apache.org/jira/browse/NIFI-5406 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Koji Kawamura >Assignee: Koji Kawamura >Priority: Major > > Current List processors (ListFile, ListFTP, ListSFTP ... etc) implementation > relies on file last modified timestamp to pick new or updated files. This > approach is efficient and lightweight in terms of state management, because > it only tracks latest modified timestamp and last executed timestamp. > However, timestamps do not work as expected in some file systems, causing > List processors missing files periodically. See NIFI-3332 comments for > details. > In order to pick every entity that has not seen before or has been updated > since it had seen last time, we need another set of processors using > different approach, that is by tracking listed entities: > * Add new abstract processor AbstractWatchEntries similar to > AbstractListProcessor but uses different approach > * Target entities have: name (path), size and last-modified-timestamp > * Implementation Processors have following properties: > ** 'Watch Time Window' to limit the maximum time period to hold the already > listed entries. E.g. if set as '30min', the processor keeps entities listed > in the last 30 mins. > ** 'Minimum File Age' to defer listing entities potentially being written > * Any entity added but not listed ever having last-modified-timestamp older > than configured 'Watch Time Window' will not be listed. If user needs to pick > these items, they have to make 'Watch Time Window' longer. It also increases > the size of data the processor has to persist in the K/V store. Efficiency vs > reliability trade-off. > * The already-listed entities are persisted into one of supported K/V store > through DistributedMapCacheClient service. User can chose what KVS to use > from HBase, Redis, Couchbase and File (DistributedMapCacheServer with > persistence file). > * The reason to use KVS instead of ManagedState is, to avoid hammering > Zookeeper too much with frequently updating Zk node with large amount of > data. The number of already-listed entries can be huge depending on > use-cases. Also, we can compress entities with DistributedMapCacheClient as > it supports putting byte array, while ManagedState only supports Map String>. > * On each onTrigger: > ** Processor performs listing. Listed entries meeting any of the following > condition will be written to the 'success' output FlowFile: > *** Not exists in the already-listed entities > *** Having newer last-modified-timestamp > *** Having different size > ** Already listed entries those are old enough compared to 'Watch Time > Window' are discarded from the already-listed entries. > * Initial supporting target is Local file system, FTP and SFTP -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2876: NIFI-5406: Adding new listing strategy to List processors.
Github user markap14 commented on the issue: https://github.com/apache/nifi/pull/2876 @ijokarumawak thanks for the updates! Code looks good! I will run a few tests and if all looks good will merge to master. Thanks! ---
[jira] [Commented] (NIFI-5436) TestRunner should allow setting the processor name
[ https://issues.apache.org/jira/browse/NIFI-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546901#comment-16546901 ] ASF GitHub Bot commented on NIFI-5436: -- Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/2902 Reviewing... > TestRunner should allow setting the processor name > -- > > Key: NIFI-5436 > URL: https://issues.apache.org/jira/browse/NIFI-5436 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > > Currently, the ProcessContext.getName() just returns a blank String. Some > users use that value of the context, but cannot test their processors > properly because there is no way to set this value. > > The TestRunner down to the MockProcessContext should support setting the name > of the processor when you create the TestRunner instance. > > If no name is provided, the the Processor class name should be used. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2902: NIFI-5436 add ability to set the processor name in the Pro...
Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/2902 Reviewing... ---
[GitHub] nifi pull request #2902: NIFI-5436 add ability to set the processor name in ...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/nifi/pull/2902#discussion_r203102528 --- Diff: nifi-docs/src/main/asciidoc/developer-guide.adoc --- @@ -2144,9 +2144,10 @@ you can use the Maven dependency: -We create a new `TestRunner` by calling the static `newTestRunner` method of the `TestRunners` class -(located in the `org.apache.nifi.util` package). This method takes a single argument. That argument can -either be the class of the Processor to test or can be an instance of a Processor. +We create a new `TestRunner` by calling on of the static `newTestRunner` methods of the `TestRunners` class --- End diff -- oops. sorry. fixed. ---
[jira] [Commented] (NIFI-5436) TestRunner should allow setting the processor name
[ https://issues.apache.org/jira/browse/NIFI-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546892#comment-16546892 ] ASF GitHub Bot commented on NIFI-5436: -- Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/nifi/pull/2902#discussion_r203102528 --- Diff: nifi-docs/src/main/asciidoc/developer-guide.adoc --- @@ -2144,9 +2144,10 @@ you can use the Maven dependency: -We create a new `TestRunner` by calling the static `newTestRunner` method of the `TestRunners` class -(located in the `org.apache.nifi.util` package). This method takes a single argument. That argument can -either be the class of the Processor to test or can be an instance of a Processor. +We create a new `TestRunner` by calling on of the static `newTestRunner` methods of the `TestRunners` class --- End diff -- oops. sorry. fixed. > TestRunner should allow setting the processor name > -- > > Key: NIFI-5436 > URL: https://issues.apache.org/jira/browse/NIFI-5436 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > > Currently, the ProcessContext.getName() just returns a blank String. Some > users use that value of the context, but cannot test their processors > properly because there is no way to set this value. > > The TestRunner down to the MockProcessContext should support setting the name > of the processor when you create the TestRunner instance. > > If no name is provided, the the Processor class name should be used. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5436) TestRunner should allow setting the processor name
[ https://issues.apache.org/jira/browse/NIFI-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546886#comment-16546886 ] ASF GitHub Bot commented on NIFI-5436: -- Github user alopresto commented on a diff in the pull request: https://github.com/apache/nifi/pull/2902#discussion_r203100791 --- Diff: nifi-docs/src/main/asciidoc/developer-guide.adoc --- @@ -2144,9 +2144,10 @@ you can use the Maven dependency: -We create a new `TestRunner` by calling the static `newTestRunner` method of the `TestRunners` class -(located in the `org.apache.nifi.util` package). This method takes a single argument. That argument can -either be the class of the Processor to test or can be an instance of a Processor. +We create a new `TestRunner` by calling on of the static `newTestRunner` methods of the `TestRunners` class --- End diff -- "calling on of" -> "calling one of" > TestRunner should allow setting the processor name > -- > > Key: NIFI-5436 > URL: https://issues.apache.org/jira/browse/NIFI-5436 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > > Currently, the ProcessContext.getName() just returns a blank String. Some > users use that value of the context, but cannot test their processors > properly because there is no way to set this value. > > The TestRunner down to the MockProcessContext should support setting the name > of the processor when you create the TestRunner instance. > > If no name is provided, the the Processor class name should be used. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #2902: NIFI-5436 add ability to set the processor name in ...
Github user alopresto commented on a diff in the pull request: https://github.com/apache/nifi/pull/2902#discussion_r203100791 --- Diff: nifi-docs/src/main/asciidoc/developer-guide.adoc --- @@ -2144,9 +2144,10 @@ you can use the Maven dependency: -We create a new `TestRunner` by calling the static `newTestRunner` method of the `TestRunners` class -(located in the `org.apache.nifi.util` package). This method takes a single argument. That argument can -either be the class of the Processor to test or can be an instance of a Processor. +We create a new `TestRunner` by calling on of the static `newTestRunner` methods of the `TestRunners` class --- End diff -- "calling on of" -> "calling one of" ---
[jira] [Created] (MINIFICPP-559) cURL built-in build fails in cross-compilation setup
Andrew Christianson created MINIFICPP-559: - Summary: cURL built-in build fails in cross-compilation setup Key: MINIFICPP-559 URL: https://issues.apache.org/jira/browse/MINIFICPP-559 Project: NiFi MiNiFi C++ Issue Type: Bug Reporter: Andrew Christianson Assignee: Andrew Christianson cURL internal build & package find fails when cross compiling. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4246) OAuth 2 Authorization support - Client Credentials Grant
[
https://issues.apache.org/jira/browse/NIFI-4246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546877#comment-16546877
]
ASF GitHub Bot commented on NIFI-4246:
--
Github user alopresto commented on the issue:
https://github.com/apache/nifi/pull/2901
Hi @jasonrogena . Thanks for submitting this. I'm sure @jdye64 appreciates
it as well because I know he is busy but wants this feature incorporated. I am
a little busy with some other things right now but I will definitely take a
look at this.
A couple quick suggestions I know I will have:
* Some of your files still reference NiFi `1.5.0-SNAPSHOT` or `1.6.0`.
Others correctly reference `1.8.0-SNAPSHOT` as the current development version.
Please ensure that all development versions are `1.8.0-SNAPSHOT`, and the
latest released version at this time is `1.7.1`.
* Some files have the license text block after package declarations,
imports, or XML opening tags. In each of these files, it needs to be the very
first block of text. You can identify issues by running the maven build with
`-Pcontrib-check` and it will highlight these issues for you. This will also
run the NiFi checkstyle rules to help identify any code style issues. If you
have issues with binary files being caught, you can exclude them in the
`pom.xml` (search the project for `` in
other `pom.xml` files for examples).
> OAuth 2 Authorization support - Client Credentials Grant
>
>
> Key: NIFI-4246
> URL: https://issues.apache.org/jira/browse/NIFI-4246
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Jeremy Dyer
>Assignee: Jeremy Dyer
>Priority: Major
>
> If your interacting with REST endpoints on the web chances are you are going
> to run into an OAuth2 secured webservice. The IETF (Internet Engineering Task
> Force) defines 4 methods in which OAuth2 authorization can occur. This JIRA
> is focused solely on the Client Credentials Grant method defined at
> https://tools.ietf.org/html/rfc6749#section-4.4
> This implementation should provide a ControllerService in which the enduser
> can configure the credentials for obtaining the authorization grant (access
> token) from the resource owner. In turn a new property will be added to the
> InvokeHTTP processor (if it doesn't already exist from one of the other JIRA
> efforts similar to this one) where the processor can reference this
> controller service to obtain the access token and insert the appropriate HTTP
> header (Authorization: Bearer{access_token}) so that the InvokeHTTP processor
> can interact with the OAuth protected resources without having to worry about
> setting up the credentials for each InvokeHTTP processor saving time and
> complexity.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi issue #2901: NIFI-4246 - Client Credentials Grant based OAuth2 Controll...
Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/2901 Hi @jasonrogena . Thanks for submitting this. I'm sure @jdye64 appreciates it as well because I know he is busy but wants this feature incorporated. I am a little busy with some other things right now but I will definitely take a look at this. A couple quick suggestions I know I will have: * Some of your files still reference NiFi `1.5.0-SNAPSHOT` or `1.6.0`. Others correctly reference `1.8.0-SNAPSHOT` as the current development version. Please ensure that all development versions are `1.8.0-SNAPSHOT`, and the latest released version at this time is `1.7.1`. * Some files have the license text block after package declarations, imports, or XML opening tags. In each of these files, it needs to be the very first block of text. You can identify issues by running the maven build with `-Pcontrib-check` and it will highlight these issues for you. This will also run the NiFi checkstyle rules to help identify any code style issues. If you have issues with binary files being caught, you can exclude them in the `pom.xml` (search the project for `` in other `pom.xml` files for examples). ---
[jira] [Updated] (NIFI-5437) Yield in ExecuteScript when catching ScriptException
[
https://issues.apache.org/jira/browse/NIFI-5437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pierre Villard updated NIFI-5437:
-
Status: Patch Available (was: Open)
> Yield in ExecuteScript when catching ScriptException
>
>
> Key: NIFI-5437
> URL: https://issues.apache.org/jira/browse/NIFI-5437
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Major
>
> In case of ScriptException in ExecuteScript, a flow file can be processed
> again and again which can lead to resource exhaustion and node instability.
> Proposition: yield the processor when an exception is caught before throwing
> a Process Exception.
> Example of exception:
> {code:java}
> 2018-07-17 16:15:23,392 ERROR [Timer-Driven Process Thread-9]
> o.a.nifi.processors.script.ExecuteScript
> ExecuteScript[id=9d776ec4-e9d7-1bca--be6c5019] Failed to process
> session due to org.apache.nifi.processor.exception.ProcessException:
> javax.script.ScriptException: TypeError: unsupported operand type(s) for +:
> 'int' and 'str' in
[jira] [Commented] (NIFI-5437) Yield in ExecuteScript when catching ScriptException
[
https://issues.apache.org/jira/browse/NIFI-5437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546852#comment-16546852
]
ASF GitHub Bot commented on NIFI-5437:
--
GitHub user pvillard31 opened a pull request:
https://github.com/apache/nifi/pull/2903
NIFI-5437 - Yield in ExecuteScript when catching ScriptException
Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with NIFI- where is the JIRA number
you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [ ] Has your PR been rebased against the latest commit within the target
branch (typically master)?
- [ ] Is your initial contribution a single, squashed commit?
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file, including the main
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in
which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/pvillard31/nifi NIFI-5437
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/2903.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2903
commit 94d2751801f9d4e6f118c89da357047e885fb850
Author: Pierre Villard
Date: 2018-07-17T16:29:32Z
NIFI-5437 - Yield in ExecuteScript when catching ScriptException
> Yield in ExecuteScript when catching ScriptException
>
>
> Key: NIFI-5437
> URL: https://issues.apache.org/jira/browse/NIFI-5437
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Major
>
> In case of ScriptException in ExecuteScript, a flow file can be processed
> again and again which can lead to resource exhaustion and node instability.
> Proposition: yield the processor when an exception is caught before throwing
> a Process Exception.
> Example of exception:
> {code:java}
> 2018-07-17 16:15:23,392 ERROR [Timer-Driven Process Thread-9]
> o.a.nifi.processors.script.ExecuteScript
> ExecuteScript[id=9d776ec4-e9d7-1bca--be6c5019] Failed to process
> session due to org.apache.nifi.processor.exception.ProcessException:
> javax.script.ScriptException: TypeError: unsupported operand type(s) for +:
> 'int' and 'str' in
[GitHub] nifi pull request #2903: NIFI-5437 - Yield in ExecuteScript when catching Sc...
GitHub user pvillard31 opened a pull request: https://github.com/apache/nifi/pull/2903 NIFI-5437 - Yield in ExecuteScript when catching ScriptException Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/pvillard31/nifi NIFI-5437 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2903.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2903 commit 94d2751801f9d4e6f118c89da357047e885fb850 Author: Pierre Villard Date: 2018-07-17T16:29:32Z NIFI-5437 - Yield in ExecuteScript when catching ScriptException ---
[jira] [Commented] (NIFI-5432) Add Syslog Record Reader legacy Syslog
[
https://issues.apache.org/jira/browse/NIFI-5432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546820#comment-16546820
]
ASF GitHub Bot commented on NIFI-5432:
--
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2900#discussion_r203081471
--- Diff:
nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/syslog/SyslogReader.java
---
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.syslog;
+
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.schema.access.SchemaAccessStrategy;
+import org.apache.nifi.schema.access.SchemaField;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.schemaregistry.services.SchemaRegistry;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.SchemaRegistryService;
+import org.apache.nifi.serialization.SimpleRecordSchema;
+import org.apache.nifi.serialization.record.RecordField;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.serialization.record.SchemaIdentifier;
+import org.apache.nifi.serialization.record.StandardSchemaIdentifier;
+import org.apache.nifi.syslog.attributes.SyslogAttributes;
+import org.apache.nifi.syslog.parsers.SyslogParser;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.EnumSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+@Tags({"syslog", "logs", "logfiles", "parse", "text", "record", "reader"})
+@CapabilityDescription("Attempts to parses the contents of a Syslog
message in accordance to RFC5424 and RFC3164. In the " +
+"case of RFC5424 formatted messages, structured data is not
supported, and will be returned as part of the message." +
+"Note: Be mindfull that RFC3164 is informational and a wide range
of different implementations are present in" +
+" the wild.")
+public class SyslogReader extends SchemaRegistryService implements
RecordReaderFactory {
+public static final String GENERIC_SYSLOG_SCHEMA_NAME =
"default-syslog-schema";
+static final AllowableValue GENERIC_SYSLOG_SCHEMA = new
AllowableValue(GENERIC_SYSLOG_SCHEMA_NAME, "Use Generic Syslog Schema",
+"The schema will be the default Syslog schema.");
+public static final PropertyDescriptor CHARSET = new
PropertyDescriptor.Builder()
+.name("Character Set")
+.description("Specifies which character set of the Syslog
messages")
+.required(true)
+.defaultValue("UTF-8")
+.addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
+.build();
+
+private volatile SyslogParser parser;
+private volatile RecordSchema recordSchema;
+
+@Override
+protected List getSupportedPropertyDescriptors() {
+final List properties = new ArrayList<>(1);
+properties.add(CHARSET);
+return properties;
+}
+
+
+@OnEnabled
+public void onEnabled(final ConfigurationContext con
[GitHub] nifi pull request #2900: NIFI-5432 Add Syslog Record Reader legacy Syslog
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2900#discussion_r203081471
--- Diff:
nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/syslog/SyslogReader.java
---
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.syslog;
+
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.schema.access.SchemaAccessStrategy;
+import org.apache.nifi.schema.access.SchemaField;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.schemaregistry.services.SchemaRegistry;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.SchemaRegistryService;
+import org.apache.nifi.serialization.SimpleRecordSchema;
+import org.apache.nifi.serialization.record.RecordField;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.serialization.record.SchemaIdentifier;
+import org.apache.nifi.serialization.record.StandardSchemaIdentifier;
+import org.apache.nifi.syslog.attributes.SyslogAttributes;
+import org.apache.nifi.syslog.parsers.SyslogParser;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.EnumSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+@Tags({"syslog", "logs", "logfiles", "parse", "text", "record", "reader"})
+@CapabilityDescription("Attempts to parses the contents of a Syslog
message in accordance to RFC5424 and RFC3164. In the " +
+"case of RFC5424 formatted messages, structured data is not
supported, and will be returned as part of the message." +
+"Note: Be mindfull that RFC3164 is informational and a wide range
of different implementations are present in" +
+" the wild.")
+public class SyslogReader extends SchemaRegistryService implements
RecordReaderFactory {
+public static final String GENERIC_SYSLOG_SCHEMA_NAME =
"default-syslog-schema";
+static final AllowableValue GENERIC_SYSLOG_SCHEMA = new
AllowableValue(GENERIC_SYSLOG_SCHEMA_NAME, "Use Generic Syslog Schema",
+"The schema will be the default Syslog schema.");
+public static final PropertyDescriptor CHARSET = new
PropertyDescriptor.Builder()
+.name("Character Set")
+.description("Specifies which character set of the Syslog
messages")
+.required(true)
+.defaultValue("UTF-8")
+.addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
+.build();
+
+private volatile SyslogParser parser;
+private volatile RecordSchema recordSchema;
+
+@Override
+protected List getSupportedPropertyDescriptors() {
+final List properties = new ArrayList<>(1);
+properties.add(CHARSET);
+return properties;
+}
+
+
+@OnEnabled
+public void onEnabled(final ConfigurationContext context) {
+final String charsetName = context.getProperty(CHARSET).getValue();
+parser = new SyslogParser(Charset.forName(charsetName));
+recordSchema = createRecordSchema();
+}
+
+@Override
+
[jira] [Created] (NIFI-5437) Yield in ExecuteScript when catching ScriptException
Pierre Villard created NIFI-5437:
Summary: Yield in ExecuteScript when catching ScriptException
Key: NIFI-5437
URL: https://issues.apache.org/jira/browse/NIFI-5437
Project: Apache NiFi
Issue Type: Improvement
Components: Extensions
Reporter: Pierre Villard
Assignee: Pierre Villard
In case of ScriptException in ExecuteScript, a flow file can be processed again
and again which can lead to resource exhaustion and node instability.
Proposition: yield the processor when an exception is caught before throwing a
Process Exception.
Example of exception:
{code:java}
2018-07-17 16:15:23,392 ERROR [Timer-Driven Process Thread-9]
o.a.nifi.processors.script.ExecuteScript
ExecuteScript[id=9d776ec4-e9d7-1bca--be6c5019] Failed to process
session due to org.apache.nifi.processor.exception.ProcessException:
javax.script.ScriptException: TypeError: unsupported operand type(s) for +:
'int' and 'str' in
[jira] [Resolved] (MINIFICPP-483) Add bustache build docs
[ https://issues.apache.org/jira/browse/MINIFICPP-483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Christianson resolved MINIFICPP-483. --- Resolution: Fixed > Add bustache build docs > --- > > Key: MINIFICPP-483 > URL: https://issues.apache.org/jira/browse/MINIFICPP-483 > Project: NiFi MiNiFi C++ > Issue Type: Documentation >Reporter: Andrew Christianson >Assignee: Andrew Christianson >Priority: Major > > Bustache adds some build-time caveats and this should be noted/clarified in > the build instructions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (MINIFICPP-459) Ensure that FlexLexer.h can be included when system includes are not in path
[ https://issues.apache.org/jira/browse/MINIFICPP-459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Christianson resolved MINIFICPP-459. --- Resolution: Fixed > Ensure that FlexLexer.h can be included when system includes are not in path > > > Key: MINIFICPP-459 > URL: https://issues.apache.org/jira/browse/MINIFICPP-459 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Andrew Christianson >Assignee: Andrew Christianson >Priority: Major > > Sources generated by bison/flex for expression language depend on lex > headers. In certain build environments, such as Android cross-compilation, > system headers are not in the include path, so the build fails due to not > finding FlexLexer.h. > Provide some way for the build to find the system FlexLexer.h corresponding > to the install of flex used to generate EL lexer sources. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5436) TestRunner should allow setting the processor name
[ https://issues.apache.org/jira/browse/NIFI-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546787#comment-16546787 ] ASF GitHub Bot commented on NIFI-5436: -- GitHub user ottobackwards opened a pull request: https://github.com/apache/nifi/pull/2902 NIFI-5436 add ability to set the processor name in the ProcessContext when usin… …g the TestRunner Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [-] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [-] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [-] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [-] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ottobackwards/nifi mock-proc-name Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2902.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2902 commit 9e967bec08714dfa0b8f17bb5348063c21be6d11 Author: Otto Fowler Date: 2018-07-17T15:26:32Z add ability to set the processor name in the ProcessContext when using the TestRunner > TestRunner should allow setting the processor name > -- > > Key: NIFI-5436 > URL: https://issues.apache.org/jira/browse/NIFI-5436 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > > Currently, the ProcessContext.getName() just returns a blank String. Some > users use that value of the context, but cannot test their processors > properly because there is no way to set this value. > > The TestRunner down to the MockProcessContext should support setting the name > of the processor when you create the TestRunner instance. > > If no name is provided, the the Processor class name should be used. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5436) TestRunner should allow setting the processor name
[ https://issues.apache.org/jira/browse/NIFI-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546788#comment-16546788 ] ASF GitHub Bot commented on NIFI-5436: -- Github user ottobackwards commented on the issue: https://github.com/apache/nifi/pull/2902 @bbende > TestRunner should allow setting the processor name > -- > > Key: NIFI-5436 > URL: https://issues.apache.org/jira/browse/NIFI-5436 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > > Currently, the ProcessContext.getName() just returns a blank String. Some > users use that value of the context, but cannot test their processors > properly because there is no way to set this value. > > The TestRunner down to the MockProcessContext should support setting the name > of the processor when you create the TestRunner instance. > > If no name is provided, the the Processor class name should be used. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2902: NIFI-5436 add ability to set the processor name in the Pro...
Github user ottobackwards commented on the issue: https://github.com/apache/nifi/pull/2902 @bbende ---
[GitHub] nifi pull request #2902: NIFI-5436 add ability to set the processor name in ...
GitHub user ottobackwards opened a pull request: https://github.com/apache/nifi/pull/2902 NIFI-5436 add ability to set the processor name in the ProcessContext when usin⦠â¦g the TestRunner Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [-] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [-] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [-] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [-] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ottobackwards/nifi mock-proc-name Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2902.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2902 commit 9e967bec08714dfa0b8f17bb5348063c21be6d11 Author: Otto Fowler Date: 2018-07-17T15:26:32Z add ability to set the processor name in the ProcessContext when using the TestRunner ---
[jira] [Commented] (NIFI-5432) Add Syslog Record Reader legacy Syslog
[
https://issues.apache.org/jira/browse/NIFI-5432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546761#comment-16546761
]
ASF GitHub Bot commented on NIFI-5432:
--
Github user bbende commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2900#discussion_r203060708
--- Diff:
nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/syslog/SyslogReader.java
---
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.syslog;
+
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.schema.access.SchemaAccessStrategy;
+import org.apache.nifi.schema.access.SchemaField;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.schemaregistry.services.SchemaRegistry;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.SchemaRegistryService;
+import org.apache.nifi.serialization.SimpleRecordSchema;
+import org.apache.nifi.serialization.record.RecordField;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.serialization.record.SchemaIdentifier;
+import org.apache.nifi.serialization.record.StandardSchemaIdentifier;
+import org.apache.nifi.syslog.attributes.SyslogAttributes;
+import org.apache.nifi.syslog.parsers.SyslogParser;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.EnumSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+@Tags({"syslog", "logs", "logfiles", "parse", "text", "record", "reader"})
+@CapabilityDescription("Attempts to parses the contents of a Syslog
message in accordance to RFC5424 and RFC3164. In the " +
+"case of RFC5424 formatted messages, structured data is not
supported, and will be returned as part of the message." +
+"Note: Be mindfull that RFC3164 is informational and a wide range
of different implementations are present in" +
+" the wild.")
+public class SyslogReader extends SchemaRegistryService implements
RecordReaderFactory {
+public static final String GENERIC_SYSLOG_SCHEMA_NAME =
"default-syslog-schema";
+static final AllowableValue GENERIC_SYSLOG_SCHEMA = new
AllowableValue(GENERIC_SYSLOG_SCHEMA_NAME, "Use Generic Syslog Schema",
+"The schema will be the default Syslog schema.");
+public static final PropertyDescriptor CHARSET = new
PropertyDescriptor.Builder()
+.name("Character Set")
+.description("Specifies which character set of the Syslog
messages")
+.required(true)
+.defaultValue("UTF-8")
+.addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
+.build();
+
+private volatile SyslogParser parser;
+private volatile RecordSchema recordSchema;
+
+@Override
+protected List getSupportedPropertyDescriptors() {
+final List properties = new ArrayList<>(1);
+properties.add(CHARSET);
+return properties;
+}
+
+
+@OnEnabled
+public void onEnabled(final ConfigurationContext context) {
[GitHub] nifi pull request #2900: NIFI-5432 Add Syslog Record Reader legacy Syslog
Github user bbende commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2900#discussion_r203060708
--- Diff:
nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/syslog/SyslogReader.java
---
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.syslog;
+
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.schema.access.SchemaAccessStrategy;
+import org.apache.nifi.schema.access.SchemaField;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.schemaregistry.services.SchemaRegistry;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.SchemaRegistryService;
+import org.apache.nifi.serialization.SimpleRecordSchema;
+import org.apache.nifi.serialization.record.RecordField;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.serialization.record.SchemaIdentifier;
+import org.apache.nifi.serialization.record.StandardSchemaIdentifier;
+import org.apache.nifi.syslog.attributes.SyslogAttributes;
+import org.apache.nifi.syslog.parsers.SyslogParser;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.EnumSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+@Tags({"syslog", "logs", "logfiles", "parse", "text", "record", "reader"})
+@CapabilityDescription("Attempts to parses the contents of a Syslog
message in accordance to RFC5424 and RFC3164. In the " +
+"case of RFC5424 formatted messages, structured data is not
supported, and will be returned as part of the message." +
+"Note: Be mindfull that RFC3164 is informational and a wide range
of different implementations are present in" +
+" the wild.")
+public class SyslogReader extends SchemaRegistryService implements
RecordReaderFactory {
+public static final String GENERIC_SYSLOG_SCHEMA_NAME =
"default-syslog-schema";
+static final AllowableValue GENERIC_SYSLOG_SCHEMA = new
AllowableValue(GENERIC_SYSLOG_SCHEMA_NAME, "Use Generic Syslog Schema",
+"The schema will be the default Syslog schema.");
+public static final PropertyDescriptor CHARSET = new
PropertyDescriptor.Builder()
+.name("Character Set")
+.description("Specifies which character set of the Syslog
messages")
+.required(true)
+.defaultValue("UTF-8")
+.addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
+.build();
+
+private volatile SyslogParser parser;
+private volatile RecordSchema recordSchema;
+
+@Override
+protected List getSupportedPropertyDescriptors() {
+final List properties = new ArrayList<>(1);
+properties.add(CHARSET);
+return properties;
+}
+
+
+@OnEnabled
+public void onEnabled(final ConfigurationContext context) {
+final String charsetName = context.getProperty(CHARSET).getValue();
+parser = new SyslogParser(Charset.forName(charsetName));
+recordSchema = createRecordSchema();
+}
+
+@Override
+protect
[jira] [Commented] (NIFI-5433) Global Menu Doesn't open on Chrome
[ https://issues.apache.org/jira/browse/NIFI-5433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546756#comment-16546756 ] Matt Gilman commented on NIFI-5433: --- Also, someone else reported similar behavior. They noticed the issue went away once Chrome updated itself. However, it's possible that there was still some cache issue happening. However, those are usually accompanied with other errors to the Dev Tools console due to incompatibility between different scripts. > Global Menu Doesn't open on Chrome > -- > > Key: NIFI-5433 > URL: https://issues.apache.org/jira/browse/NIFI-5433 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.7.0 >Reporter: Mark Jackson >Priority: Minor > > After installing nifi 1.7.0, you start up nifi (no changes). Then you click > the global menu at the top right (the kebab menu; aka the one with the 3 > horizontal lines) and nothing happens. There are no messages in the console > or any network requests. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MINIFICPP-483) Add bustache build docs
[ https://issues.apache.org/jira/browse/MINIFICPP-483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546744#comment-16546744 ] ASF GitHub Bot commented on MINIFICPP-483: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi-minifi-cpp/pull/369 > Add bustache build docs > --- > > Key: MINIFICPP-483 > URL: https://issues.apache.org/jira/browse/MINIFICPP-483 > Project: NiFi MiNiFi C++ > Issue Type: Documentation >Reporter: Andrew Christianson >Assignee: Andrew Christianson >Priority: Major > > Bustache adds some build-time caveats and this should be noted/clarified in > the build instructions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MINIFICPP-459) Ensure that FlexLexer.h can be included when system includes are not in path
[ https://issues.apache.org/jira/browse/MINIFICPP-459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546743#comment-16546743 ] ASF GitHub Bot commented on MINIFICPP-459: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi-minifi-cpp/pull/370 > Ensure that FlexLexer.h can be included when system includes are not in path > > > Key: MINIFICPP-459 > URL: https://issues.apache.org/jira/browse/MINIFICPP-459 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Andrew Christianson >Assignee: Andrew Christianson >Priority: Major > > Sources generated by bison/flex for expression language depend on lex > headers. In certain build environments, such as Android cross-compilation, > system headers are not in the include path, so the build fails due to not > finding FlexLexer.h. > Provide some way for the build to find the system FlexLexer.h corresponding > to the install of flex used to generate EL lexer sources. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-5436) TestRunner should allow setting the processor name
Otto Fowler created NIFI-5436: - Summary: TestRunner should allow setting the processor name Key: NIFI-5436 URL: https://issues.apache.org/jira/browse/NIFI-5436 Project: Apache NiFi Issue Type: Improvement Reporter: Otto Fowler Assignee: Otto Fowler Currently, the ProcessContext.getName() just returns a blank String. Some users use that value of the context, but cannot test their processors properly because there is no way to set this value. The TestRunner down to the MockProcessContext should support setting the name of the processor when you create the TestRunner instance. If no name is provided, the the Processor class name should be used. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #369: MINIFICPP-483 Added compile-time bustache...
Github user asfgit closed the pull request at: https://github.com/apache/nifi-minifi-cpp/pull/369 ---
[GitHub] nifi-minifi-cpp pull request #370: MINIFICPP-459 Added build-time caveat for...
Github user asfgit closed the pull request at: https://github.com/apache/nifi-minifi-cpp/pull/370 ---
[jira] [Commented] (NIFI-5432) Add Syslog Record Reader legacy Syslog
[
https://issues.apache.org/jira/browse/NIFI-5432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546737#comment-16546737
]
ASF GitHub Bot commented on NIFI-5432:
--
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2900#discussion_r203054640
--- Diff:
nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/syslog/SyslogReader.java
---
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.syslog;
+
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.schema.access.SchemaAccessStrategy;
+import org.apache.nifi.schema.access.SchemaField;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.schemaregistry.services.SchemaRegistry;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.SchemaRegistryService;
+import org.apache.nifi.serialization.SimpleRecordSchema;
+import org.apache.nifi.serialization.record.RecordField;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.serialization.record.SchemaIdentifier;
+import org.apache.nifi.serialization.record.StandardSchemaIdentifier;
+import org.apache.nifi.syslog.attributes.SyslogAttributes;
+import org.apache.nifi.syslog.parsers.SyslogParser;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.EnumSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+@Tags({"syslog", "logs", "logfiles", "parse", "text", "record", "reader"})
+@CapabilityDescription("Attempts to parses the contents of a Syslog
message in accordance to RFC5424 and RFC3164. In the " +
+"case of RFC5424 formatted messages, structured data is not
supported, and will be returned as part of the message." +
+"Note: Be mindfull that RFC3164 is informational and a wide range
of different implementations are present in" +
+" the wild.")
+public class SyslogReader extends SchemaRegistryService implements
RecordReaderFactory {
+public static final String GENERIC_SYSLOG_SCHEMA_NAME =
"default-syslog-schema";
+static final AllowableValue GENERIC_SYSLOG_SCHEMA = new
AllowableValue(GENERIC_SYSLOG_SCHEMA_NAME, "Use Generic Syslog Schema",
+"The schema will be the default Syslog schema.");
+public static final PropertyDescriptor CHARSET = new
PropertyDescriptor.Builder()
+.name("Character Set")
+.description("Specifies which character set of the Syslog
messages")
+.required(true)
+.defaultValue("UTF-8")
+.addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
+.build();
+
+private volatile SyslogParser parser;
+private volatile RecordSchema recordSchema;
+
+@Override
+protected List getSupportedPropertyDescriptors() {
+final List properties = new ArrayList<>(1);
+properties.add(CHARSET);
+return properties;
+}
+
+
+@OnEnabled
+public void onEnabled(final ConfigurationContext con
[GitHub] nifi pull request #2900: NIFI-5432 Add Syslog Record Reader legacy Syslog
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2900#discussion_r203054640
--- Diff:
nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/syslog/SyslogReader.java
---
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.syslog;
+
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.schema.access.SchemaAccessStrategy;
+import org.apache.nifi.schema.access.SchemaField;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.schemaregistry.services.SchemaRegistry;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.SchemaRegistryService;
+import org.apache.nifi.serialization.SimpleRecordSchema;
+import org.apache.nifi.serialization.record.RecordField;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.serialization.record.SchemaIdentifier;
+import org.apache.nifi.serialization.record.StandardSchemaIdentifier;
+import org.apache.nifi.syslog.attributes.SyslogAttributes;
+import org.apache.nifi.syslog.parsers.SyslogParser;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.EnumSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+@Tags({"syslog", "logs", "logfiles", "parse", "text", "record", "reader"})
+@CapabilityDescription("Attempts to parses the contents of a Syslog
message in accordance to RFC5424 and RFC3164. In the " +
+"case of RFC5424 formatted messages, structured data is not
supported, and will be returned as part of the message." +
+"Note: Be mindfull that RFC3164 is informational and a wide range
of different implementations are present in" +
+" the wild.")
+public class SyslogReader extends SchemaRegistryService implements
RecordReaderFactory {
+public static final String GENERIC_SYSLOG_SCHEMA_NAME =
"default-syslog-schema";
+static final AllowableValue GENERIC_SYSLOG_SCHEMA = new
AllowableValue(GENERIC_SYSLOG_SCHEMA_NAME, "Use Generic Syslog Schema",
+"The schema will be the default Syslog schema.");
+public static final PropertyDescriptor CHARSET = new
PropertyDescriptor.Builder()
+.name("Character Set")
+.description("Specifies which character set of the Syslog
messages")
+.required(true)
+.defaultValue("UTF-8")
+.addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
+.build();
+
+private volatile SyslogParser parser;
+private volatile RecordSchema recordSchema;
+
+@Override
+protected List getSupportedPropertyDescriptors() {
+final List properties = new ArrayList<>(1);
+properties.add(CHARSET);
+return properties;
+}
+
+
+@OnEnabled
+public void onEnabled(final ConfigurationContext context) {
+final String charsetName = context.getProperty(CHARSET).getValue();
+parser = new SyslogParser(Charset.forName(charsetName));
+recordSchema = createRecordSchema();
+}
+
+@Override
+
[jira] [Commented] (NIFI-5432) Add Syslog Record Reader legacy Syslog
[
https://issues.apache.org/jira/browse/NIFI-5432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546733#comment-16546733
]
ASF GitHub Bot commented on NIFI-5432:
--
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2900#discussion_r203054280
--- Diff:
nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/syslog/SyslogReader.java
---
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.syslog;
+
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.schema.access.SchemaAccessStrategy;
+import org.apache.nifi.schema.access.SchemaField;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.schemaregistry.services.SchemaRegistry;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.SchemaRegistryService;
+import org.apache.nifi.serialization.SimpleRecordSchema;
+import org.apache.nifi.serialization.record.RecordField;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.serialization.record.SchemaIdentifier;
+import org.apache.nifi.serialization.record.StandardSchemaIdentifier;
+import org.apache.nifi.syslog.attributes.SyslogAttributes;
+import org.apache.nifi.syslog.parsers.SyslogParser;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.EnumSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+@Tags({"syslog", "logs", "logfiles", "parse", "text", "record", "reader"})
+@CapabilityDescription("Attempts to parses the contents of a Syslog
message in accordance to RFC5424 and RFC3164. In the " +
+"case of RFC5424 formatted messages, structured data is not
supported, and will be returned as part of the message." +
+"Note: Be mindfull that RFC3164 is informational and a wide range
of different implementations are present in" +
+" the wild.")
+public class SyslogReader extends SchemaRegistryService implements
RecordReaderFactory {
+public static final String GENERIC_SYSLOG_SCHEMA_NAME =
"default-syslog-schema";
+static final AllowableValue GENERIC_SYSLOG_SCHEMA = new
AllowableValue(GENERIC_SYSLOG_SCHEMA_NAME, "Use Generic Syslog Schema",
+"The schema will be the default Syslog schema.");
+public static final PropertyDescriptor CHARSET = new
PropertyDescriptor.Builder()
+.name("Character Set")
+.description("Specifies which character set of the Syslog
messages")
+.required(true)
+.defaultValue("UTF-8")
+.addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
+.build();
+
+private volatile SyslogParser parser;
+private volatile RecordSchema recordSchema;
+
+@Override
+protected List getSupportedPropertyDescriptors() {
+final List properties = new ArrayList<>(1);
+properties.add(CHARSET);
+return properties;
+}
+
+
+@OnEnabled
+public void onEnabled(final ConfigurationContext con
[GitHub] nifi pull request #2900: NIFI-5432 Add Syslog Record Reader legacy Syslog
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2900#discussion_r203054280
--- Diff:
nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/syslog/SyslogReader.java
---
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.syslog;
+
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.schema.access.SchemaAccessStrategy;
+import org.apache.nifi.schema.access.SchemaField;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.schemaregistry.services.SchemaRegistry;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.SchemaRegistryService;
+import org.apache.nifi.serialization.SimpleRecordSchema;
+import org.apache.nifi.serialization.record.RecordField;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.serialization.record.SchemaIdentifier;
+import org.apache.nifi.serialization.record.StandardSchemaIdentifier;
+import org.apache.nifi.syslog.attributes.SyslogAttributes;
+import org.apache.nifi.syslog.parsers.SyslogParser;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.EnumSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+@Tags({"syslog", "logs", "logfiles", "parse", "text", "record", "reader"})
+@CapabilityDescription("Attempts to parses the contents of a Syslog
message in accordance to RFC5424 and RFC3164. In the " +
+"case of RFC5424 formatted messages, structured data is not
supported, and will be returned as part of the message." +
+"Note: Be mindfull that RFC3164 is informational and a wide range
of different implementations are present in" +
+" the wild.")
+public class SyslogReader extends SchemaRegistryService implements
RecordReaderFactory {
+public static final String GENERIC_SYSLOG_SCHEMA_NAME =
"default-syslog-schema";
+static final AllowableValue GENERIC_SYSLOG_SCHEMA = new
AllowableValue(GENERIC_SYSLOG_SCHEMA_NAME, "Use Generic Syslog Schema",
+"The schema will be the default Syslog schema.");
+public static final PropertyDescriptor CHARSET = new
PropertyDescriptor.Builder()
+.name("Character Set")
+.description("Specifies which character set of the Syslog
messages")
+.required(true)
+.defaultValue("UTF-8")
+.addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
+.build();
+
+private volatile SyslogParser parser;
+private volatile RecordSchema recordSchema;
+
+@Override
+protected List getSupportedPropertyDescriptors() {
+final List properties = new ArrayList<>(1);
+properties.add(CHARSET);
+return properties;
+}
+
+
+@OnEnabled
+public void onEnabled(final ConfigurationContext context) {
+final String charsetName = context.getProperty(CHARSET).getValue();
+parser = new SyslogParser(Charset.forName(charsetName));
+recordSchema = createRecordSchema();
+}
+
+@Override
+
[jira] [Commented] (NIFI-5432) Add Syslog Record Reader legacy Syslog
[
https://issues.apache.org/jira/browse/NIFI-5432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546728#comment-16546728
]
ASF GitHub Bot commented on NIFI-5432:
--
Github user bbende commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2900#discussion_r203052373
--- Diff:
nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/syslog/SyslogReader.java
---
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.syslog;
+
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.schema.access.SchemaAccessStrategy;
+import org.apache.nifi.schema.access.SchemaField;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.schemaregistry.services.SchemaRegistry;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.SchemaRegistryService;
+import org.apache.nifi.serialization.SimpleRecordSchema;
+import org.apache.nifi.serialization.record.RecordField;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.serialization.record.SchemaIdentifier;
+import org.apache.nifi.serialization.record.StandardSchemaIdentifier;
+import org.apache.nifi.syslog.attributes.SyslogAttributes;
+import org.apache.nifi.syslog.parsers.SyslogParser;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.EnumSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+@Tags({"syslog", "logs", "logfiles", "parse", "text", "record", "reader"})
+@CapabilityDescription("Attempts to parses the contents of a Syslog
message in accordance to RFC5424 and RFC3164. In the " +
+"case of RFC5424 formatted messages, structured data is not
supported, and will be returned as part of the message." +
+"Note: Be mindfull that RFC3164 is informational and a wide range
of different implementations are present in" +
+" the wild.")
+public class SyslogReader extends SchemaRegistryService implements
RecordReaderFactory {
+public static final String GENERIC_SYSLOG_SCHEMA_NAME =
"default-syslog-schema";
+static final AllowableValue GENERIC_SYSLOG_SCHEMA = new
AllowableValue(GENERIC_SYSLOG_SCHEMA_NAME, "Use Generic Syslog Schema",
+"The schema will be the default Syslog schema.");
+public static final PropertyDescriptor CHARSET = new
PropertyDescriptor.Builder()
+.name("Character Set")
+.description("Specifies which character set of the Syslog
messages")
+.required(true)
+.defaultValue("UTF-8")
+.addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
+.build();
+
+private volatile SyslogParser parser;
+private volatile RecordSchema recordSchema;
+
+@Override
+protected List getSupportedPropertyDescriptors() {
+final List properties = new ArrayList<>(1);
+properties.add(CHARSET);
+return properties;
+}
+
+
+@OnEnabled
+public void onEnabled(final ConfigurationContext context) {
[GitHub] nifi pull request #2900: NIFI-5432 Add Syslog Record Reader legacy Syslog
Github user bbende commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2900#discussion_r203052373
--- Diff:
nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/syslog/SyslogReader.java
---
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.syslog;
+
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.schema.access.SchemaAccessStrategy;
+import org.apache.nifi.schema.access.SchemaField;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.schemaregistry.services.SchemaRegistry;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.SchemaRegistryService;
+import org.apache.nifi.serialization.SimpleRecordSchema;
+import org.apache.nifi.serialization.record.RecordField;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.serialization.record.SchemaIdentifier;
+import org.apache.nifi.serialization.record.StandardSchemaIdentifier;
+import org.apache.nifi.syslog.attributes.SyslogAttributes;
+import org.apache.nifi.syslog.parsers.SyslogParser;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.EnumSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+@Tags({"syslog", "logs", "logfiles", "parse", "text", "record", "reader"})
+@CapabilityDescription("Attempts to parses the contents of a Syslog
message in accordance to RFC5424 and RFC3164. In the " +
+"case of RFC5424 formatted messages, structured data is not
supported, and will be returned as part of the message." +
+"Note: Be mindfull that RFC3164 is informational and a wide range
of different implementations are present in" +
+" the wild.")
+public class SyslogReader extends SchemaRegistryService implements
RecordReaderFactory {
+public static final String GENERIC_SYSLOG_SCHEMA_NAME =
"default-syslog-schema";
+static final AllowableValue GENERIC_SYSLOG_SCHEMA = new
AllowableValue(GENERIC_SYSLOG_SCHEMA_NAME, "Use Generic Syslog Schema",
+"The schema will be the default Syslog schema.");
+public static final PropertyDescriptor CHARSET = new
PropertyDescriptor.Builder()
+.name("Character Set")
+.description("Specifies which character set of the Syslog
messages")
+.required(true)
+.defaultValue("UTF-8")
+.addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
+.build();
+
+private volatile SyslogParser parser;
+private volatile RecordSchema recordSchema;
+
+@Override
+protected List getSupportedPropertyDescriptors() {
+final List properties = new ArrayList<>(1);
+properties.add(CHARSET);
+return properties;
+}
+
+
+@OnEnabled
+public void onEnabled(final ConfigurationContext context) {
+final String charsetName = context.getProperty(CHARSET).getValue();
+parser = new SyslogParser(Charset.forName(charsetName));
+recordSchema = createRecordSchema();
+}
+
+@Override
+protect
[jira] [Updated] (MINIFICPP-558) Move PayloadSerializer in preparation for Coap
[ https://issues.apache.org/jira/browse/MINIFICPP-558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] marco polo updated MINIFICPP-558: - Fix Version/s: 0.6.0 > Move PayloadSerializer in preparation for Coap > -- > > Key: MINIFICPP-558 > URL: https://issues.apache.org/jira/browse/MINIFICPP-558 > Project: NiFi MiNiFi C++ > Issue Type: Bug >Reporter: marco polo >Assignee: marco polo >Priority: Major > Fix For: 0.6.0 > > > Move PayloadSerializer -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (MINIFICPP-558) Move PayloadSerializer in preparation for Coap
marco polo created MINIFICPP-558: Summary: Move PayloadSerializer in preparation for Coap Key: MINIFICPP-558 URL: https://issues.apache.org/jira/browse/MINIFICPP-558 Project: NiFi MiNiFi C++ Issue Type: Bug Reporter: marco polo Assignee: marco polo Move PayloadSerializer -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (MINIFICPP-554) RocksDB initialization may interfere with agent startup.
[ https://issues.apache.org/jira/browse/MINIFICPP-554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] marco polo updated MINIFICPP-554: - Status: Patch Available (was: In Progress) > RocksDB initialization may interfere with agent startup. > - > > Key: MINIFICPP-554 > URL: https://issues.apache.org/jira/browse/MINIFICPP-554 > Project: NiFi MiNiFi C++ > Issue Type: Sub-task >Reporter: marco polo >Assignee: marco polo >Priority: Critical > Fix For: 0.6.0 > > > An arbitrarily large number of FFs within the flow file repo could prevent > startup since we're attempting to prune the internal database. This can be > done after startup. > > This is a bug that can cause problems for agents starting up. > > Steps that I'm working on now: Checkpoint the DB and then we can > asynchronously deal with the old flow files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (MINIFICPP-554) RocksDB initialization may interfere with agent startup.
[ https://issues.apache.org/jira/browse/MINIFICPP-554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] marco polo updated MINIFICPP-554: - Fix Version/s: 0.6.0 > RocksDB initialization may interfere with agent startup. > - > > Key: MINIFICPP-554 > URL: https://issues.apache.org/jira/browse/MINIFICPP-554 > Project: NiFi MiNiFi C++ > Issue Type: Sub-task >Reporter: marco polo >Assignee: marco polo >Priority: Critical > Fix For: 0.6.0 > > > An arbitrarily large number of FFs within the flow file repo could prevent > startup since we're attempting to prune the internal database. This can be > done after startup. > > This is a bug that can cause problems for agents starting up. > > Steps that I'm working on now: Checkpoint the DB and then we can > asynchronously deal with the old flow files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MINIFICPP-554) RocksDB initialization may interfere with agent startup.
[ https://issues.apache.org/jira/browse/MINIFICPP-554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546656#comment-16546656 ] ASF GitHub Bot commented on MINIFICPP-554: -- GitHub user phrocker opened a pull request: https://github.com/apache/nifi-minifi-cpp/pull/373 MINIFICPP-554: Add checkpointing to the startup process to delete flow files to avoid startup problems. Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/phrocker/nifi-minifi-cpp MINIFICPP-554 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-minifi-cpp/pull/373.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #373 commit 1e0ea6df94cb0075f9f718b1d26a18de942d7e3f Author: Marc Parisi Date: 2018-07-16T12:13:13Z MINIFICPP-554: Add checkpointing to the startup process to delete flow files to avoid startup problems. > RocksDB initialization may interfere with agent startup. > - > > Key: MINIFICPP-554 > URL: https://issues.apache.org/jira/browse/MINIFICPP-554 > Project: NiFi MiNiFi C++ > Issue Type: Sub-task >Reporter: marco polo >Assignee: marco polo >Priority: Critical > > An arbitrarily large number of FFs within the flow file repo could prevent > startup since we're attempting to prune the internal database. This can be > done after startup. > > This is a bug that can cause problems for agents starting up. > > Steps that I'm working on now: Checkpoint the DB and then we can > asynchronously deal with the old flow files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #373: MINIFICPP-554: Add checkpointing to the s...
GitHub user phrocker opened a pull request: https://github.com/apache/nifi-minifi-cpp/pull/373 MINIFICPP-554: Add checkpointing to the startup process to delete flow files to avoid startup problems. Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/phrocker/nifi-minifi-cpp MINIFICPP-554 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-minifi-cpp/pull/373.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #373 commit 1e0ea6df94cb0075f9f718b1d26a18de942d7e3f Author: Marc Parisi Date: 2018-07-16T12:13:13Z MINIFICPP-554: Add checkpointing to the startup process to delete flow files to avoid startup problems. ---
[jira] [Created] (NIFI-5435) Prometheus /metrics http endpoint for monitoring integration
Hari Sekhon created NIFI-5435: - Summary: Prometheus /metrics http endpoint for monitoring integration Key: NIFI-5435 URL: https://issues.apache.org/jira/browse/NIFI-5435 Project: Apache NiFi Issue Type: New Feature Components: Tools and Build Affects Versions: 1.5.0 Reporter: Hari Sekhon Feature Request to add Prometheus /metrics http endpoint for monitoring integration: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Cscrape_config%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5434) ExecuteScript/Jython imports fail randomly
[
https://issues.apache.org/jira/browse/NIFI-5434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wietze B updated NIFI-5434:
---
Description:
I have a number of ExecuteScript processors running Jython that import
libraries such as {{threading}}, {{urllib3}}, {{requests}}, etc.
When I boot up my machine or initially start the NiFi service, the
ExecuteScript processors relying on external libraries usually to don't work
straight away, throwing {{AttributeError}} s or {{ImportError}} s on libraries
that definitely exist and are accessible. Restarting the machine or the NiFi
service again usually fixes the problem, but it is unclear to me why the
imports sometimes fail, and sometimes work.
See attached two screenshots and an extract from the logs.
was:
I have a number of ExecuteScript processors running Jython that import
libraries such as {{threading}}, {{urllib3}}, {{requests}}, etc.
When I boot up my machine or initially start the NiFi service, the
ExecuteScript processors relying on external libraries usually to don't work
straight away, throwing {{AttributeError}} s or {{ImportError}} s on libraries
that definitely exist and are accessible. Restarting the machine or the NiFi
service again usually fixes the problem, but it is unclear to me why the
imports sometime fail, and sometimes work.
See attached two screenshots and an extract from the logs.
> ExecuteScript/Jython imports fail randomly
> --
>
> Key: NIFI-5434
> URL: https://issues.apache.org/jira/browse/NIFI-5434
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.6.0, 1.7.0, 1.7.1
> Environment: Debian Stretch (9.5)
>Reporter: Wietze B
>Priority: Major
> Attachments: ibm.png, java_thread.png, log_extract.log
>
>
> I have a number of ExecuteScript processors running Jython that import
> libraries such as {{threading}}, {{urllib3}}, {{requests}}, etc.
> When I boot up my machine or initially start the NiFi service, the
> ExecuteScript processors relying on external libraries usually to don't work
> straight away, throwing {{AttributeError}} s or {{ImportError}} s on
> libraries that definitely exist and are accessible. Restarting the machine or
> the NiFi service again usually fixes the problem, but it is unclear to me why
> the imports sometimes fail, and sometimes work.
> See attached two screenshots and an extract from the logs.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5427) ScrollElasticsearchHttp breaks with Elasticsearch 6
[ https://issues.apache.org/jira/browse/NIFI-5427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546621#comment-16546621 ] ASF GitHub Bot commented on NIFI-5427: -- Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2890 (Will merge if it works against an ES6 docker image) > ScrollElasticsearchHttp breaks with Elasticsearch 6 > --- > > Key: NIFI-5427 > URL: https://issues.apache.org/jira/browse/NIFI-5427 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.4.0, 1.5.0, 1.6.0, 1.7.0 >Reporter: Joseph Gresock >Assignee: Joseph Gresock >Priority: Minor > > The Elasticsearch 6 Scroll API uses scroll IDs that are nearly 10,000 > characters long. Since the ScrollElasticsearchHttp processor uses a GET > request for scrolling, this breaks the processor after the first scroll call. > > The solution is fairly simple: use a POST instead of a GET for the scroll > calls. This is backwards compatible down to Elasticsearch 2.X. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
