Re: Issues with creating and connecting RPG programmatically

[Matt Gilman]

I've responded to the question on StackOverflow [1].

Thanks

Matt

[1]
http://stackoverflow.com/questions/43788780/nifi-issues-with-creating-and-connecting-rpg-programmatically/43789310#43789310

On Thu, May 4, 2017 at 12:49 PM, Pushkara R  wrote:

> Hi,
>
> I am trying to write a function that takes two parameters, a processor ID
> in machine A and an input port in machine B and creates an RPG in machine B
> which connects the above to.
>
> Now, how I am doing that is
> 1. A POST to the
> */nifi-api/process-groups//remote-process-groups*
> endpoint to create an RPG and retrieve the ID of that RPG
> 2. A POST to the */nifi-api/process-groups//connections
> *endpoint to create a connection between the processor and the input
> port. the processorID and the ID of the input port are being provided along
> with the list of the relationships.
> 3. A final PUT to */nifi-api/remote-process-groups/ *to enable the
> transmission between the machines*.*
>
> Now, the function always throws errors in step 2. A 409 is thrown for the
> POST request with the error being 'Unable to find specified destination'.
> (though refreshing the canvas on machine 1 shows the rpg having been
> created)
> However, when I manually run the steps 2 and 3 afterwards, with the same
> rpgid, the connection happens.
>
> Now, I'm not sure if this is a synchronization issue or not, but I want to
> figure it out because I would not want to separate out steps 1 2 and 3.
> Could somebody point out what could be the issue here?
>
> Pushkar
>
> PS - the post messages for steps 2 are the same when the api is called
> from within the function and manually.
>

Re: Closing in on a NiFi 1.2.0 release?

[Bryan Bende]

The issues from yesterday have been resolved so I'll start kicking off
another attempt at the RC process.


On Wed, May 3, 2017 at 6:10 PM, Bryan Bende  wrote:

> Quick update... I ran into two issues that will need to be addressed to
> create the RC.
>
> I've created JIRAs for them and tagged them as 1.2:
>
> https://issues.apache.org/jira/browse/NIFI-3795
> https://issues.apache.org/jira/browse/NIFI-3793
>
>
> On Wed, May 3, 2017 at 2:41 PM, Bryan Bende  wrote:
>
>> Looks like all of the JIRAs have been resolved and we are in a good place.
>>
>> I'll begin kicking off the RC process.
>>
>> On Tue, May 2, 2017 at 5:48 PM, Andre  wrote:
>>
>>> All,
>>>
>>> For some reason my canvas did not refresh after a process bounce (which
>>> generally occurs) but reloading page allows for modifications.
>>>
>>> Cheers
>>>
>>> On Wed, May 3, 2017 at 7:43 AM, Andre  wrote:
>>>
 folks,

 I was just working to debug the final thorns found reviewing NIFI-3726
 and noticed an odd behavior and wanted to confirm.

 If I recall correctly in the past users could simply replace a
 processor NAR file and even if that NAR existed the flow would continue to
 work.

 I just replaced

 cp ~/nifi/nifi-nar-bundles/nifi-cybersecurity-bundle/nifi-cyber
 security-nar/target/nifi-cybersecurity-nar-1.2.0-SNAPSHOT.nar
 ~/devel/nifi-1.2.0-SNAPSHOT/lib/nifi-cybersecurity-nar-1.2.0
 -SNAPSHOT.nar

 (note the different ~/nifi ~/devel used to ensure I don't explode the
 rest of the already compiled components).

 When I try to make changes to the flow I am displayed with the
 following error:

 [image: Inline image 1]

 This happens even when I try to drag and drop connected processors
 around the canvas.


 Oddly enough I can still add and delete components to the canvas but
 whatever touches the tainted processor cannot be modified at all.

 Examples of messages:

 *Attempt to move*

 Component Position
 [5, cb0a31ac-015b-1000-7473-873a47eb702e,
 cb0a52ab-015b-1000-e43a-f6293a9ae99d] is not the most up-to-date
 revision. This component appears to have been modified


 *Attempt to delete a downstream processor*
 Error
 [1, cb0a31ac-015b-1000-7473-873a47eb702e,
 cb0b2ae4-015b-1000-35a8-9eaf6a45fc6a] is not the most up-to-date
 revision. This component appears to have been modified


 I don't have a 1.1.0 instance around me at the moment but I vaguely
 remember being able to do that in the past.

 Can someone confirm this is new and expected behavior?

 Cheers


 On Wed, May 3, 2017 at 5:54 AM, Andy LoPresto 
 wrote:

> I’ll review & merge as soon as they are available.
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On May 2, 2017, at 3:51 PM, Bryan Bende  wrote:
>
> Thanks Drew. These seem like good candidates for the release.
>
> On Tue, May 2, 2017 at 3:42 PM, Andrew Lim 
> wrote:
>
> There are three doc updates/additions that would be great to include
> in the RC:
>
> https://issues.apache.org/jira/browse/NIFI-3701
> https://issues.apache.org/jira/browse/NIFI-3773
> https://issues.apache.org/jira/browse/NIFI-3774
>
> Sarah Olson and I have been working on these.  We should have PRs
> submitted for them very soon.
>
> -Drew
>
>
> On May 2, 2017, at 2:11 PM, Aldrin Piri  wrote:
>
> Haven't had much luck in getting our Docker efforts incorporated into
> Docker Hub.  As a result I have created an issue to track that
> integration
> [1] and resolved the original issue.
>
> We can evaluate our options and figure out the best path forward.  At
> this
> time procedures are not yet well established within ASF to support
> configuring these builds.
>
> [1] https://issues.apache.org/jira/browse/NIFI-3772
>
> On Tue, May 2, 2017 at 11:13 AM, Andrew Lim <
> andrewlim.apa...@gmail.com>
> wrote:
>
> I will be making updates to the Release Notes and Migration Guidance
> doc
> regarding the TLS 1.2 version support.  Tracked by:
>
> https://issues.apache.org/jira/browse/NIFI-3720
>
>
> -Drew
>
>
> On May 2, 2017, at 11:08 AM, Joe Witt  wrote:
>
> Those are great updates.  I'd recommend we avoid highlighting the
> versions of UI components though.
>
> Thanks
>
>
> On Tue, May 2, 2017 at 11:03 AM, Scott Aslan 
>
> wrote:
>
> Hey 

Issues with creating and connecting RPG programmatically

[Pushkara R]

Hi,

I am trying to write a function that takes two parameters, a processor ID
in machine A and an input port in machine B and creates an RPG in machine B
which connects the above to.

Now, how I am doing that is
1. A POST to the
*/nifi-api/process-groups//remote-process-groups*
endpoint to create an RPG and retrieve the ID of that RPG
2. A POST to the */nifi-api/process-groups//connections
*endpoint to create a connection between the processor and the input port.
the processorID and the ID of the input port are being provided along with
the list of the relationships.
3. A final PUT to */nifi-api/remote-process-groups/ *to enable the
transmission between the machines*.*

Now, the function always throws errors in step 2. A 409 is thrown for the
POST request with the error being 'Unable to find specified destination'.
(though refreshing the canvas on machine 1 shows the rpg having been
created)
However, when I manually run the steps 2 and 3 afterwards, with the same
rpgid, the connection happens.

Now, I'm not sure if this is a synchronization issue or not, but I want to
figure it out because I would not want to separate out steps 1 2 and 3.
Could somebody point out what could be the issue here?

Pushkar

PS - the post messages for steps 2 are the same when the api is called from
within the function and manually.

Re: MiNiFi C++ Expression Language

[Matt Burgess]

Good point, we should select a parser generator that produces tight code. 
Having said that, the MVP grammar is so simple that it could easily be put into 
a parser generator DSL later, and the MVP implementation could be hand-written 
for the first iteration (I'm picturing a couple dozen lines of code, but that's 
just a knee-jerk estimation).


> On May 4, 2017, at 11:25 AM, Marc  wrote:
> 
> Andy,
>Depending on the type of parser chosen you may run into memory
> limitations on devices that run MiNiFi-C++. Since that's a deployment
> concern -- and one that few will face, I would add that we can make this a
> conditional build and only including the necessary components within the
> binary if a build option is not specified to exclude these components.
> 
> On Thu, May 4, 2017 at 11:14 AM, Andrew Christianson <
> andrew.christian...@nextcentury.com> wrote:
> 
>> The island grammar sounds very appealing for an MVP. Simple to implement,
>> yet it covers a very common EL use case (dynamically inserting attr vals
>> into property values). If we have general consensus I would love to see a
>> MINIFI JIRA ticket added for this.
>> 
>> -Andy
>> 
>> From: Matt Burgess 
>> Sent: Thursday, May 4, 2017 11:09:46 AM
>> To: dev@nifi.apache.org
>> Subject: Re: MiNiFi C++ Expression Language
>> 
>> What you have sounds good to me. IMO minimum viable product would be an
>> island grammar (meaning you can have any characters outside a ${}
>> expression) and inside would support an attribute name. Next steps could be
>> nested expressions and/or support for functions, added piecemeal as the
>> contributor sees fit. I wasn't involved in the early development of the EL
>> grammar, so maybe someone who was has some thoughts on a natural evolution.
>> 
>> Regards,
>> Matt
>> 
>>> On May 4, 2017, at 11:00 AM, Andrew Christianson > nextcentury.com> wrote:
>>> 
>>> My bad, what does the sketch of the plan *look like*?
>>> 
>>> -Andy
>>> 
>>> From: Andrew Christianson
>>> Sent: Thursday, May 4, 2017 10:59:07 AM
>>> To: dev@nifi.apache.org
>>> Subject: Re: MiNiFi C++ Expression Language
>>> 
>>> What does the sketch of the plan to do the separate implementation?
>> Write a flex/bison grammar, hook it into the cmake build, and start using
>> it? Any constraints on features or syntax that this separate implementation
>> must support?
>>> 
>>> -Andy
>>> 
>>> From: Matt Burgess 
>>> Sent: Thursday, May 4, 2017 9:37:34 AM
>>> To: dev@nifi.apache.org
>>> Subject: Re: MiNiFi C++ Expression Language
>>> 
>>> No plans that I know of. In the meantime, EL support for MiNiFi is kind
>> of held hostage, so maybe the separate implementation is more viable in the
>> nearer term. If/When the ANTLR4 upgrade happens, we could replace whatever
>> exists by then with the cross-platform ANTLR target generation, and test
>> the whole kit and caboodle.
>>> 
>>> Anyone out there familiar with the aforementioned tools (or willing to
>> hand-roll one)? What do you think about this "EL bootstrapping" approach?
>>> 
>>> Thanks,
>>> Matt
>>> 
 On May 4, 2017, at 9:22 AM, Andrew Christianson > nextcentury.com> wrote:
 
 Got it. So the crux of the problem is porting from v3 to v4, plus the
>> added uncertainty of the C++ v4 target.
 
 I'm assuming that NiFi wants to eventually get onto v4 anyway. If
>> that's the case, then porting to v4 is probably the ticket. Are there any
>> concrete plans to do so in the NiFi mother project yet?
 
 -Andy
 
 From: Matt Burgess 
 Sent: Thursday, May 4, 2017 9:18:00 AM
 To: dev@nifi.apache.org
 Subject: Re: MiNiFi C++ Expression Language
 
 Correct, the current NiFi EL grammar is ANTLR3.
 
 
> On May 4, 2017, at 9:12 AM, Andrew Christianson > nextcentury.com> wrote:
> 
> Do I understand correctly that NiFi is currently using ANTLRv3?
> 
> From: Matt Burgess 
> Sent: Thursday, May 4, 2017 9:05:35 AM
> To: dev@nifi.apache.org
> Subject: Re: MiNiFi C++ Expression Language
> 
> I haven't used Flex/Bison since a trivial example in college, so I'm
>> not sure about the LOE for getting that set up, maybe there's a Maven-built
>> project out there that we could look at for inspiration, but that seems
>> unlikely :)
> 
> An ANTLR4 refactor (assuming the C++ target is in good shape) would
>> give us NiFi/MiNiFi EL compatibility (and full-featured EL support in
>> MiNiFi C++), but we'd have to accept the risks of introducing bugs,
>> regressions, etc. as a result of the refactor. Basically we'd just need to
>> test the heck out of it on all platforms, which 

Re: MiNiFi C++ Expression Language

[Marc]

Andy,
Depending on the type of parser chosen you may run into memory
limitations on devices that run MiNiFi-C++. Since that's a deployment
concern -- and one that few will face, I would add that we can make this a
conditional build and only including the necessary components within the
binary if a build option is not specified to exclude these components.

On Thu, May 4, 2017 at 11:14 AM, Andrew Christianson <
andrew.christian...@nextcentury.com> wrote:

> The island grammar sounds very appealing for an MVP. Simple to implement,
> yet it covers a very common EL use case (dynamically inserting attr vals
> into property values). If we have general consensus I would love to see a
> MINIFI JIRA ticket added for this.
>
> -Andy
> 
> From: Matt Burgess 
> Sent: Thursday, May 4, 2017 11:09:46 AM
> To: dev@nifi.apache.org
> Subject: Re: MiNiFi C++ Expression Language
>
> What you have sounds good to me. IMO minimum viable product would be an
> island grammar (meaning you can have any characters outside a ${}
> expression) and inside would support an attribute name. Next steps could be
> nested expressions and/or support for functions, added piecemeal as the
> contributor sees fit. I wasn't involved in the early development of the EL
> grammar, so maybe someone who was has some thoughts on a natural evolution.
>
> Regards,
> Matt
>
> > On May 4, 2017, at 11:00 AM, Andrew Christianson  nextcentury.com> wrote:
> >
> > My bad, what does the sketch of the plan *look like*?
> >
> > -Andy
> > 
> > From: Andrew Christianson
> > Sent: Thursday, May 4, 2017 10:59:07 AM
> > To: dev@nifi.apache.org
> > Subject: Re: MiNiFi C++ Expression Language
> >
> > What does the sketch of the plan to do the separate implementation?
> Write a flex/bison grammar, hook it into the cmake build, and start using
> it? Any constraints on features or syntax that this separate implementation
> must support?
> >
> > -Andy
> > 
> > From: Matt Burgess 
> > Sent: Thursday, May 4, 2017 9:37:34 AM
> > To: dev@nifi.apache.org
> > Subject: Re: MiNiFi C++ Expression Language
> >
> > No plans that I know of. In the meantime, EL support for MiNiFi is kind
> of held hostage, so maybe the separate implementation is more viable in the
> nearer term. If/When the ANTLR4 upgrade happens, we could replace whatever
> exists by then with the cross-platform ANTLR target generation, and test
> the whole kit and caboodle.
> >
> > Anyone out there familiar with the aforementioned tools (or willing to
> hand-roll one)? What do you think about this "EL bootstrapping" approach?
> >
> > Thanks,
> > Matt
> >
> >> On May 4, 2017, at 9:22 AM, Andrew Christianson  nextcentury.com> wrote:
> >>
> >> Got it. So the crux of the problem is porting from v3 to v4, plus the
> added uncertainty of the C++ v4 target.
> >>
> >> I'm assuming that NiFi wants to eventually get onto v4 anyway. If
> that's the case, then porting to v4 is probably the ticket. Are there any
> concrete plans to do so in the NiFi mother project yet?
> >>
> >> -Andy
> >> 
> >> From: Matt Burgess 
> >> Sent: Thursday, May 4, 2017 9:18:00 AM
> >> To: dev@nifi.apache.org
> >> Subject: Re: MiNiFi C++ Expression Language
> >>
> >> Correct, the current NiFi EL grammar is ANTLR3.
> >>
> >>
> >>> On May 4, 2017, at 9:12 AM, Andrew Christianson  nextcentury.com> wrote:
> >>>
> >>> Do I understand correctly that NiFi is currently using ANTLRv3?
> >>> 
> >>> From: Matt Burgess 
> >>> Sent: Thursday, May 4, 2017 9:05:35 AM
> >>> To: dev@nifi.apache.org
> >>> Subject: Re: MiNiFi C++ Expression Language
> >>>
> >>> I haven't used Flex/Bison since a trivial example in college, so I'm
> not sure about the LOE for getting that set up, maybe there's a Maven-built
> project out there that we could look at for inspiration, but that seems
> unlikely :)
> >>>
> >>> An ANTLR4 refactor (assuming the C++ target is in good shape) would
> give us NiFi/MiNiFi EL compatibility (and full-featured EL support in
> MiNiFi C++), but we'd have to accept the risks of introducing bugs,
> regressions, etc. as a result of the refactor. Basically we'd just need to
> test the heck out of it on all platforms, which isn't a bad thing but adds
> to the LOE for the ANTLR4 upgrade, versus a smaller testing "surface" for
> incremental development of a C/C++ based grammar.
> >>>
> >>>
> >>> On May 4, 2017, at 8:51 AM, Andrew Christianson  nextcentury.com> wrote:
> >>>
> > I tried a quick ANTLR4 upgrade myself, it's indeed a big job to
> refactor the existing grammar. Since the source and target for MiNiFi C++
> is, well, C++, an alternative could be to use lex/yacc (Flex/Bison) [1],
> 

Re: MiNiFi C++ Expression Language

[Andrew Christianson]

The island grammar sounds very appealing for an MVP. Simple to implement, yet 
it covers a very common EL use case (dynamically inserting attr vals into 
property values). If we have general consensus I would love to see a MINIFI 
JIRA ticket added for this.

-Andy

From: Matt Burgess 
Sent: Thursday, May 4, 2017 11:09:46 AM
To: dev@nifi.apache.org
Subject: Re: MiNiFi C++ Expression Language

What you have sounds good to me. IMO minimum viable product would be an island 
grammar (meaning you can have any characters outside a ${} expression) and 
inside would support an attribute name. Next steps could be nested expressions 
and/or support for functions, added piecemeal as the contributor sees fit. I 
wasn't involved in the early development of the EL grammar, so maybe someone 
who was has some thoughts on a natural evolution.

Regards,
Matt

> On May 4, 2017, at 11:00 AM, Andrew Christianson 
>  wrote:
>
> My bad, what does the sketch of the plan *look like*?
>
> -Andy
> 
> From: Andrew Christianson
> Sent: Thursday, May 4, 2017 10:59:07 AM
> To: dev@nifi.apache.org
> Subject: Re: MiNiFi C++ Expression Language
>
> What does the sketch of the plan to do the separate implementation? Write a 
> flex/bison grammar, hook it into the cmake build, and start using it? Any 
> constraints on features or syntax that this separate implementation must 
> support?
>
> -Andy
> 
> From: Matt Burgess 
> Sent: Thursday, May 4, 2017 9:37:34 AM
> To: dev@nifi.apache.org
> Subject: Re: MiNiFi C++ Expression Language
>
> No plans that I know of. In the meantime, EL support for MiNiFi is kind of 
> held hostage, so maybe the separate implementation is more viable in the 
> nearer term. If/When the ANTLR4 upgrade happens, we could replace whatever 
> exists by then with the cross-platform ANTLR target generation, and test the 
> whole kit and caboodle.
>
> Anyone out there familiar with the aforementioned tools (or willing to 
> hand-roll one)? What do you think about this "EL bootstrapping" approach?
>
> Thanks,
> Matt
>
>> On May 4, 2017, at 9:22 AM, Andrew Christianson 
>>  wrote:
>>
>> Got it. So the crux of the problem is porting from v3 to v4, plus the added 
>> uncertainty of the C++ v4 target.
>>
>> I'm assuming that NiFi wants to eventually get onto v4 anyway. If that's the 
>> case, then porting to v4 is probably the ticket. Are there any concrete 
>> plans to do so in the NiFi mother project yet?
>>
>> -Andy
>> 
>> From: Matt Burgess 
>> Sent: Thursday, May 4, 2017 9:18:00 AM
>> To: dev@nifi.apache.org
>> Subject: Re: MiNiFi C++ Expression Language
>>
>> Correct, the current NiFi EL grammar is ANTLR3.
>>
>>
>>> On May 4, 2017, at 9:12 AM, Andrew Christianson 
>>>  wrote:
>>>
>>> Do I understand correctly that NiFi is currently using ANTLRv3?
>>> 
>>> From: Matt Burgess 
>>> Sent: Thursday, May 4, 2017 9:05:35 AM
>>> To: dev@nifi.apache.org
>>> Subject: Re: MiNiFi C++ Expression Language
>>>
>>> I haven't used Flex/Bison since a trivial example in college, so I'm not 
>>> sure about the LOE for getting that set up, maybe there's a Maven-built 
>>> project out there that we could look at for inspiration, but that seems 
>>> unlikely :)
>>>
>>> An ANTLR4 refactor (assuming the C++ target is in good shape) would give us 
>>> NiFi/MiNiFi EL compatibility (and full-featured EL support in MiNiFi C++), 
>>> but we'd have to accept the risks of introducing bugs, regressions, etc. as 
>>> a result of the refactor. Basically we'd just need to test the heck out of 
>>> it on all platforms, which isn't a bad thing but adds to the LOE for the 
>>> ANTLR4 upgrade, versus a smaller testing "surface" for incremental 
>>> development of a C/C++ based grammar.
>>>
>>>
>>> On May 4, 2017, at 8:51 AM, Andrew Christianson 
>>>  wrote:
>>>
> I tried a quick ANTLR4 upgrade myself, it's indeed a big job to refactor 
> the existing grammar. Since the source and target for MiNiFi C++ is, 
> well, C++, an alternative could be to use lex/yacc (Flex/Bison) [1], 
> Lemon [2], Ragel [3], etc. The downside is maintaining two grammars, but 
> we are doing that with all the MiNiFi components already. The upside is 
> being able to support EL incrementally as the grammar is developed. What 
> do you think?

 This seems like a pragmatic approach. What's the level-of-effort required 
 to do the initial grammar port and set up the build tooling? Less than 
 refactoring for ANTLR4? I'm not as familiar with the EL grammar situation.

 -Andy
 

Re: MiNiFi C++ Expression Language

[Matt Burgess]

What you have sounds good to me. IMO minimum viable product would be an island 
grammar (meaning you can have any characters outside a ${} expression) and 
inside would support an attribute name. Next steps could be nested expressions 
and/or support for functions, added piecemeal as the contributor sees fit. I 
wasn't involved in the early development of the EL grammar, so maybe someone 
who was has some thoughts on a natural evolution.

Regards,
Matt

> On May 4, 2017, at 11:00 AM, Andrew Christianson 
>  wrote:
> 
> My bad, what does the sketch of the plan *look like*?
> 
> -Andy
> 
> From: Andrew Christianson
> Sent: Thursday, May 4, 2017 10:59:07 AM
> To: dev@nifi.apache.org
> Subject: Re: MiNiFi C++ Expression Language
> 
> What does the sketch of the plan to do the separate implementation? Write a 
> flex/bison grammar, hook it into the cmake build, and start using it? Any 
> constraints on features or syntax that this separate implementation must 
> support?
> 
> -Andy
> 
> From: Matt Burgess 
> Sent: Thursday, May 4, 2017 9:37:34 AM
> To: dev@nifi.apache.org
> Subject: Re: MiNiFi C++ Expression Language
> 
> No plans that I know of. In the meantime, EL support for MiNiFi is kind of 
> held hostage, so maybe the separate implementation is more viable in the 
> nearer term. If/When the ANTLR4 upgrade happens, we could replace whatever 
> exists by then with the cross-platform ANTLR target generation, and test the 
> whole kit and caboodle.
> 
> Anyone out there familiar with the aforementioned tools (or willing to 
> hand-roll one)? What do you think about this "EL bootstrapping" approach?
> 
> Thanks,
> Matt
> 
>> On May 4, 2017, at 9:22 AM, Andrew Christianson 
>>  wrote:
>> 
>> Got it. So the crux of the problem is porting from v3 to v4, plus the added 
>> uncertainty of the C++ v4 target.
>> 
>> I'm assuming that NiFi wants to eventually get onto v4 anyway. If that's the 
>> case, then porting to v4 is probably the ticket. Are there any concrete 
>> plans to do so in the NiFi mother project yet?
>> 
>> -Andy
>> 
>> From: Matt Burgess 
>> Sent: Thursday, May 4, 2017 9:18:00 AM
>> To: dev@nifi.apache.org
>> Subject: Re: MiNiFi C++ Expression Language
>> 
>> Correct, the current NiFi EL grammar is ANTLR3.
>> 
>> 
>>> On May 4, 2017, at 9:12 AM, Andrew Christianson 
>>>  wrote:
>>> 
>>> Do I understand correctly that NiFi is currently using ANTLRv3?
>>> 
>>> From: Matt Burgess 
>>> Sent: Thursday, May 4, 2017 9:05:35 AM
>>> To: dev@nifi.apache.org
>>> Subject: Re: MiNiFi C++ Expression Language
>>> 
>>> I haven't used Flex/Bison since a trivial example in college, so I'm not 
>>> sure about the LOE for getting that set up, maybe there's a Maven-built 
>>> project out there that we could look at for inspiration, but that seems 
>>> unlikely :)
>>> 
>>> An ANTLR4 refactor (assuming the C++ target is in good shape) would give us 
>>> NiFi/MiNiFi EL compatibility (and full-featured EL support in MiNiFi C++), 
>>> but we'd have to accept the risks of introducing bugs, regressions, etc. as 
>>> a result of the refactor. Basically we'd just need to test the heck out of 
>>> it on all platforms, which isn't a bad thing but adds to the LOE for the 
>>> ANTLR4 upgrade, versus a smaller testing "surface" for incremental 
>>> development of a C/C++ based grammar.
>>> 
>>> 
>>> On May 4, 2017, at 8:51 AM, Andrew Christianson 
>>>  wrote:
>>> 
> I tried a quick ANTLR4 upgrade myself, it's indeed a big job to refactor 
> the existing grammar. Since the source and target for MiNiFi C++ is, 
> well, C++, an alternative could be to use lex/yacc (Flex/Bison) [1], 
> Lemon [2], Ragel [3], etc. The downside is maintaining two grammars, but 
> we are doing that with all the MiNiFi components already. The upside is 
> being able to support EL incrementally as the grammar is developed. What 
> do you think?
 
 This seems like a pragmatic approach. What's the level-of-effort required 
 to do the initial grammar port and set up the build tooling? Less than 
 refactoring for ANTLR4? I'm not as familiar with the EL grammar situation.
 
 -Andy
 
 From: Matt Burgess 
 Sent: Thursday, May 4, 2017 8:46:20 AM
 To: dev@nifi.apache.org
 Subject: Re: MiNiFi C++ Expression Language
 
 I tried a quick ANTLR4 upgrade myself, it's indeed a big job to refactor 
 the existing grammar. Since the source and target for MiNiFi C++ is, well, 
 C++, an alternative could be to use lex/yacc (Flex/Bison) [1], Lemon [2], 
 Ragel [3], 

Re: MiNiFi C++ Expression Language

[Andrew Christianson]

My bad, what does the sketch of the plan *look like*?

-Andy

From: Andrew Christianson
Sent: Thursday, May 4, 2017 10:59:07 AM
To: dev@nifi.apache.org
Subject: Re: MiNiFi C++ Expression Language

What does the sketch of the plan to do the separate implementation? Write a 
flex/bison grammar, hook it into the cmake build, and start using it? Any 
constraints on features or syntax that this separate implementation must 
support?

-Andy

From: Matt Burgess 
Sent: Thursday, May 4, 2017 9:37:34 AM
To: dev@nifi.apache.org
Subject: Re: MiNiFi C++ Expression Language

No plans that I know of. In the meantime, EL support for MiNiFi is kind of held 
hostage, so maybe the separate implementation is more viable in the nearer 
term. If/When the ANTLR4 upgrade happens, we could replace whatever exists by 
then with the cross-platform ANTLR target generation, and test the whole kit 
and caboodle.

Anyone out there familiar with the aforementioned tools (or willing to 
hand-roll one)? What do you think about this "EL bootstrapping" approach?

Thanks,
Matt

> On May 4, 2017, at 9:22 AM, Andrew Christianson 
>  wrote:
>
> Got it. So the crux of the problem is porting from v3 to v4, plus the added 
> uncertainty of the C++ v4 target.
>
> I'm assuming that NiFi wants to eventually get onto v4 anyway. If that's the 
> case, then porting to v4 is probably the ticket. Are there any concrete plans 
> to do so in the NiFi mother project yet?
>
> -Andy
> 
> From: Matt Burgess 
> Sent: Thursday, May 4, 2017 9:18:00 AM
> To: dev@nifi.apache.org
> Subject: Re: MiNiFi C++ Expression Language
>
> Correct, the current NiFi EL grammar is ANTLR3.
>
>
>> On May 4, 2017, at 9:12 AM, Andrew Christianson 
>>  wrote:
>>
>> Do I understand correctly that NiFi is currently using ANTLRv3?
>> 
>> From: Matt Burgess 
>> Sent: Thursday, May 4, 2017 9:05:35 AM
>> To: dev@nifi.apache.org
>> Subject: Re: MiNiFi C++ Expression Language
>>
>> I haven't used Flex/Bison since a trivial example in college, so I'm not 
>> sure about the LOE for getting that set up, maybe there's a Maven-built 
>> project out there that we could look at for inspiration, but that seems 
>> unlikely :)
>>
>> An ANTLR4 refactor (assuming the C++ target is in good shape) would give us 
>> NiFi/MiNiFi EL compatibility (and full-featured EL support in MiNiFi C++), 
>> but we'd have to accept the risks of introducing bugs, regressions, etc. as 
>> a result of the refactor. Basically we'd just need to test the heck out of 
>> it on all platforms, which isn't a bad thing but adds to the LOE for the 
>> ANTLR4 upgrade, versus a smaller testing "surface" for incremental 
>> development of a C/C++ based grammar.
>>
>>
>> On May 4, 2017, at 8:51 AM, Andrew Christianson 
>>  wrote:
>>
 I tried a quick ANTLR4 upgrade myself, it's indeed a big job to refactor 
 the existing grammar. Since the source and target for MiNiFi C++ is, well, 
 C++, an alternative could be to use lex/yacc (Flex/Bison) [1], Lemon [2], 
 Ragel [3], etc. The downside is maintaining two grammars, but we are doing 
 that with all the MiNiFi components already. The upside is being able to 
 support EL incrementally as the grammar is developed. What do you think?
>>>
>>> This seems like a pragmatic approach. What's the level-of-effort required 
>>> to do the initial grammar port and set up the build tooling? Less than 
>>> refactoring for ANTLR4? I'm not as familiar with the EL grammar situation.
>>>
>>> -Andy
>>> 
>>> From: Matt Burgess 
>>> Sent: Thursday, May 4, 2017 8:46:20 AM
>>> To: dev@nifi.apache.org
>>> Subject: Re: MiNiFi C++ Expression Language
>>>
>>> I tried a quick ANTLR4 upgrade myself, it's indeed a big job to refactor 
>>> the existing grammar. Since the source and target for MiNiFi C++ is, well, 
>>> C++, an alternative could be to use lex/yacc (Flex/Bison) [1], Lemon [2], 
>>> Ragel [3], etc. The downside is maintaining two grammars, but we are doing 
>>> that with all the MiNiFi components already. The upside is being able to 
>>> support EL incrementally as the grammar is developed. What do you think?
>>>
>>> Regards,
>>> Matt
>>>
>>> [1] http://dinosaur.compilertools.net/
>>> [2] http://www.hwaci.com/sw/lemon/
>>> [3] http://www.colm.net/open-source/ragel/
>>>
>>>
>>>
>>> Sent from my iPhone
 On May 4, 2017, at 8:13 AM, Marc P.  wrote:

 Andrew,
 I am not aware of it being actively worked [1]. This would require using
 ANTLR4, but I don't believe C++ support is well tested [2].  Someone can
 correct me if I'm wrong, but there would have to be 

Re: MiNiFi C++ Expression Language

[Andrew Christianson]

What does the sketch of the plan to do the separate implementation? Write a 
flex/bison grammar, hook it into the cmake build, and start using it? Any 
constraints on features or syntax that this separate implementation must 
support?

-Andy

From: Matt Burgess 
Sent: Thursday, May 4, 2017 9:37:34 AM
To: dev@nifi.apache.org
Subject: Re: MiNiFi C++ Expression Language

No plans that I know of. In the meantime, EL support for MiNiFi is kind of held 
hostage, so maybe the separate implementation is more viable in the nearer 
term. If/When the ANTLR4 upgrade happens, we could replace whatever exists by 
then with the cross-platform ANTLR target generation, and test the whole kit 
and caboodle.

Anyone out there familiar with the aforementioned tools (or willing to 
hand-roll one)? What do you think about this "EL bootstrapping" approach?

Thanks,
Matt

> On May 4, 2017, at 9:22 AM, Andrew Christianson 
>  wrote:
>
> Got it. So the crux of the problem is porting from v3 to v4, plus the added 
> uncertainty of the C++ v4 target.
>
> I'm assuming that NiFi wants to eventually get onto v4 anyway. If that's the 
> case, then porting to v4 is probably the ticket. Are there any concrete plans 
> to do so in the NiFi mother project yet?
>
> -Andy
> 
> From: Matt Burgess 
> Sent: Thursday, May 4, 2017 9:18:00 AM
> To: dev@nifi.apache.org
> Subject: Re: MiNiFi C++ Expression Language
>
> Correct, the current NiFi EL grammar is ANTLR3.
>
>
>> On May 4, 2017, at 9:12 AM, Andrew Christianson 
>>  wrote:
>>
>> Do I understand correctly that NiFi is currently using ANTLRv3?
>> 
>> From: Matt Burgess 
>> Sent: Thursday, May 4, 2017 9:05:35 AM
>> To: dev@nifi.apache.org
>> Subject: Re: MiNiFi C++ Expression Language
>>
>> I haven't used Flex/Bison since a trivial example in college, so I'm not 
>> sure about the LOE for getting that set up, maybe there's a Maven-built 
>> project out there that we could look at for inspiration, but that seems 
>> unlikely :)
>>
>> An ANTLR4 refactor (assuming the C++ target is in good shape) would give us 
>> NiFi/MiNiFi EL compatibility (and full-featured EL support in MiNiFi C++), 
>> but we'd have to accept the risks of introducing bugs, regressions, etc. as 
>> a result of the refactor. Basically we'd just need to test the heck out of 
>> it on all platforms, which isn't a bad thing but adds to the LOE for the 
>> ANTLR4 upgrade, versus a smaller testing "surface" for incremental 
>> development of a C/C++ based grammar.
>>
>>
>> On May 4, 2017, at 8:51 AM, Andrew Christianson 
>>  wrote:
>>
 I tried a quick ANTLR4 upgrade myself, it's indeed a big job to refactor 
 the existing grammar. Since the source and target for MiNiFi C++ is, well, 
 C++, an alternative could be to use lex/yacc (Flex/Bison) [1], Lemon [2], 
 Ragel [3], etc. The downside is maintaining two grammars, but we are doing 
 that with all the MiNiFi components already. The upside is being able to 
 support EL incrementally as the grammar is developed. What do you think?
>>>
>>> This seems like a pragmatic approach. What's the level-of-effort required 
>>> to do the initial grammar port and set up the build tooling? Less than 
>>> refactoring for ANTLR4? I'm not as familiar with the EL grammar situation.
>>>
>>> -Andy
>>> 
>>> From: Matt Burgess 
>>> Sent: Thursday, May 4, 2017 8:46:20 AM
>>> To: dev@nifi.apache.org
>>> Subject: Re: MiNiFi C++ Expression Language
>>>
>>> I tried a quick ANTLR4 upgrade myself, it's indeed a big job to refactor 
>>> the existing grammar. Since the source and target for MiNiFi C++ is, well, 
>>> C++, an alternative could be to use lex/yacc (Flex/Bison) [1], Lemon [2], 
>>> Ragel [3], etc. The downside is maintaining two grammars, but we are doing 
>>> that with all the MiNiFi components already. The upside is being able to 
>>> support EL incrementally as the grammar is developed. What do you think?
>>>
>>> Regards,
>>> Matt
>>>
>>> [1] http://dinosaur.compilertools.net/
>>> [2] http://www.hwaci.com/sw/lemon/
>>> [3] http://www.colm.net/open-source/ragel/
>>>
>>>
>>>
>>> Sent from my iPhone
 On May 4, 2017, at 8:13 AM, Marc P.  wrote:

 Andrew,
 I am not aware of it being actively worked [1]. This would require using
 ANTLR4, but I don't believe C++ support is well tested [2].  Someone can
 correct me if I'm wrong, but there would have to be changes to both sides.
 I attempted a quick straw man with grammars, but didn't take it very far
 after making initial changes to the grammar. It generated code, but I'm
 uncertain of cross platform compatibility with the 

Re: MiNiFi C++ Expression Language

[Matt Burgess]

No plans that I know of. In the meantime, EL support for MiNiFi is kind of held 
hostage, so maybe the separate implementation is more viable in the nearer 
term. If/When the ANTLR4 upgrade happens, we could replace whatever exists by 
then with the cross-platform ANTLR target generation, and test the whole kit 
and caboodle.

Anyone out there familiar with the aforementioned tools (or willing to 
hand-roll one)? What do you think about this "EL bootstrapping" approach?

Thanks,
Matt

> On May 4, 2017, at 9:22 AM, Andrew Christianson 
>  wrote:
> 
> Got it. So the crux of the problem is porting from v3 to v4, plus the added 
> uncertainty of the C++ v4 target.
> 
> I'm assuming that NiFi wants to eventually get onto v4 anyway. If that's the 
> case, then porting to v4 is probably the ticket. Are there any concrete plans 
> to do so in the NiFi mother project yet?
> 
> -Andy
> 
> From: Matt Burgess 
> Sent: Thursday, May 4, 2017 9:18:00 AM
> To: dev@nifi.apache.org
> Subject: Re: MiNiFi C++ Expression Language
> 
> Correct, the current NiFi EL grammar is ANTLR3.
> 
> 
>> On May 4, 2017, at 9:12 AM, Andrew Christianson 
>>  wrote:
>> 
>> Do I understand correctly that NiFi is currently using ANTLRv3?
>> 
>> From: Matt Burgess 
>> Sent: Thursday, May 4, 2017 9:05:35 AM
>> To: dev@nifi.apache.org
>> Subject: Re: MiNiFi C++ Expression Language
>> 
>> I haven't used Flex/Bison since a trivial example in college, so I'm not 
>> sure about the LOE for getting that set up, maybe there's a Maven-built 
>> project out there that we could look at for inspiration, but that seems 
>> unlikely :)
>> 
>> An ANTLR4 refactor (assuming the C++ target is in good shape) would give us 
>> NiFi/MiNiFi EL compatibility (and full-featured EL support in MiNiFi C++), 
>> but we'd have to accept the risks of introducing bugs, regressions, etc. as 
>> a result of the refactor. Basically we'd just need to test the heck out of 
>> it on all platforms, which isn't a bad thing but adds to the LOE for the 
>> ANTLR4 upgrade, versus a smaller testing "surface" for incremental 
>> development of a C/C++ based grammar.
>> 
>> 
>> On May 4, 2017, at 8:51 AM, Andrew Christianson 
>>  wrote:
>> 
 I tried a quick ANTLR4 upgrade myself, it's indeed a big job to refactor 
 the existing grammar. Since the source and target for MiNiFi C++ is, well, 
 C++, an alternative could be to use lex/yacc (Flex/Bison) [1], Lemon [2], 
 Ragel [3], etc. The downside is maintaining two grammars, but we are doing 
 that with all the MiNiFi components already. The upside is being able to 
 support EL incrementally as the grammar is developed. What do you think?
>>> 
>>> This seems like a pragmatic approach. What's the level-of-effort required 
>>> to do the initial grammar port and set up the build tooling? Less than 
>>> refactoring for ANTLR4? I'm not as familiar with the EL grammar situation.
>>> 
>>> -Andy
>>> 
>>> From: Matt Burgess 
>>> Sent: Thursday, May 4, 2017 8:46:20 AM
>>> To: dev@nifi.apache.org
>>> Subject: Re: MiNiFi C++ Expression Language
>>> 
>>> I tried a quick ANTLR4 upgrade myself, it's indeed a big job to refactor 
>>> the existing grammar. Since the source and target for MiNiFi C++ is, well, 
>>> C++, an alternative could be to use lex/yacc (Flex/Bison) [1], Lemon [2], 
>>> Ragel [3], etc. The downside is maintaining two grammars, but we are doing 
>>> that with all the MiNiFi components already. The upside is being able to 
>>> support EL incrementally as the grammar is developed. What do you think?
>>> 
>>> Regards,
>>> Matt
>>> 
>>> [1] http://dinosaur.compilertools.net/
>>> [2] http://www.hwaci.com/sw/lemon/
>>> [3] http://www.colm.net/open-source/ragel/
>>> 
>>> 
>>> 
>>> Sent from my iPhone
 On May 4, 2017, at 8:13 AM, Marc P.  wrote:
 
 Andrew,
 I am not aware of it being actively worked [1]. This would require using
 ANTLR4, but I don't believe C++ support is well tested [2].  Someone can
 correct me if I'm wrong, but there would have to be changes to both sides.
 I attempted a quick straw man with grammars, but didn't take it very far
 after making initial changes to the grammar. It generated code, but I'm
 uncertain of cross platform compatibility with the expression language. If
 that's not expected or required that will remove some limitations as a
 result of moving to ANTLR4.
 
 [1] https://issues.apache.org/jira/browse/MINIFI-140
 [2] http://www.soft-gems.net/index.php/tools/49-the-antlr4-c-target-is-here
 
 On Thu, May 4, 2017 at 8:07 AM, Andrew Christianson <
 andrew.christian...@nextcentury.com> wrote:
 
> All,
> 

Re: MiNiFi C++ Expression Language

[Andrew Christianson]

Got it. So the crux of the problem is porting from v3 to v4, plus the added 
uncertainty of the C++ v4 target.

I'm assuming that NiFi wants to eventually get onto v4 anyway. If that's the 
case, then porting to v4 is probably the ticket. Are there any concrete plans 
to do so in the NiFi mother project yet?

-Andy

From: Matt Burgess 
Sent: Thursday, May 4, 2017 9:18:00 AM
To: dev@nifi.apache.org
Subject: Re: MiNiFi C++ Expression Language

Correct, the current NiFi EL grammar is ANTLR3.


> On May 4, 2017, at 9:12 AM, Andrew Christianson 
>  wrote:
>
> Do I understand correctly that NiFi is currently using ANTLRv3?
> 
> From: Matt Burgess 
> Sent: Thursday, May 4, 2017 9:05:35 AM
> To: dev@nifi.apache.org
> Subject: Re: MiNiFi C++ Expression Language
>
> I haven't used Flex/Bison since a trivial example in college, so I'm not sure 
> about the LOE for getting that set up, maybe there's a Maven-built project 
> out there that we could look at for inspiration, but that seems unlikely :)
>
> An ANTLR4 refactor (assuming the C++ target is in good shape) would give us 
> NiFi/MiNiFi EL compatibility (and full-featured EL support in MiNiFi C++), 
> but we'd have to accept the risks of introducing bugs, regressions, etc. as a 
> result of the refactor. Basically we'd just need to test the heck out of it 
> on all platforms, which isn't a bad thing but adds to the LOE for the ANTLR4 
> upgrade, versus a smaller testing "surface" for incremental development of a 
> C/C++ based grammar.
>
>
> On May 4, 2017, at 8:51 AM, Andrew Christianson 
>  wrote:
>
>>> I tried a quick ANTLR4 upgrade myself, it's indeed a big job to refactor 
>>> the existing grammar. Since the source and target for MiNiFi C++ is, well, 
>>> C++, an alternative could be to use lex/yacc (Flex/Bison) [1], Lemon [2], 
>>> Ragel [3], etc. The downside is maintaining two grammars, but we are doing 
>>> that with all the MiNiFi components already. The upside is being able to 
>>> support EL incrementally as the grammar is developed. What do you think?
>>
>> This seems like a pragmatic approach. What's the level-of-effort required to 
>> do the initial grammar port and set up the build tooling? Less than 
>> refactoring for ANTLR4? I'm not as familiar with the EL grammar situation.
>>
>> -Andy
>> 
>> From: Matt Burgess 
>> Sent: Thursday, May 4, 2017 8:46:20 AM
>> To: dev@nifi.apache.org
>> Subject: Re: MiNiFi C++ Expression Language
>>
>> I tried a quick ANTLR4 upgrade myself, it's indeed a big job to refactor the 
>> existing grammar. Since the source and target for MiNiFi C++ is, well, C++, 
>> an alternative could be to use lex/yacc (Flex/Bison) [1], Lemon [2], Ragel 
>> [3], etc. The downside is maintaining two grammars, but we are doing that 
>> with all the MiNiFi components already. The upside is being able to support 
>> EL incrementally as the grammar is developed. What do you think?
>>
>> Regards,
>> Matt
>>
>> [1] http://dinosaur.compilertools.net/
>> [2] http://www.hwaci.com/sw/lemon/
>> [3] http://www.colm.net/open-source/ragel/
>>
>>
>>
>> Sent from my iPhone
>>> On May 4, 2017, at 8:13 AM, Marc P.  wrote:
>>>
>>> Andrew,
>>> I am not aware of it being actively worked [1]. This would require using
>>> ANTLR4, but I don't believe C++ support is well tested [2].  Someone can
>>> correct me if I'm wrong, but there would have to be changes to both sides.
>>> I attempted a quick straw man with grammars, but didn't take it very far
>>> after making initial changes to the grammar. It generated code, but I'm
>>> uncertain of cross platform compatibility with the expression language. If
>>> that's not expected or required that will remove some limitations as a
>>> result of moving to ANTLR4.
>>>
>>> [1] https://issues.apache.org/jira/browse/MINIFI-140
>>> [2] http://www.soft-gems.net/index.php/tools/49-the-antlr4-c-target-is-here
>>>
>>> On Thu, May 4, 2017 at 8:07 AM, Andrew Christianson <
>>> andrew.christian...@nextcentury.com> wrote:
>>>
 All,

 I see that we do not have support for the expression language yet in
 MiNiFi C++. Is anyone actively working on this, and if so, is there an ETA?
 If no one is working on it, is there a general plan for how it should be
 implemented? I think I recall seeing references to ANTLR

Re: MiNiFi C++ Expression Language

[Matt Burgess]

Correct, the current NiFi EL grammar is ANTLR3.


> On May 4, 2017, at 9:12 AM, Andrew Christianson 
>  wrote:
> 
> Do I understand correctly that NiFi is currently using ANTLRv3?
> 
> From: Matt Burgess 
> Sent: Thursday, May 4, 2017 9:05:35 AM
> To: dev@nifi.apache.org
> Subject: Re: MiNiFi C++ Expression Language
> 
> I haven't used Flex/Bison since a trivial example in college, so I'm not sure 
> about the LOE for getting that set up, maybe there's a Maven-built project 
> out there that we could look at for inspiration, but that seems unlikely :)
> 
> An ANTLR4 refactor (assuming the C++ target is in good shape) would give us 
> NiFi/MiNiFi EL compatibility (and full-featured EL support in MiNiFi C++), 
> but we'd have to accept the risks of introducing bugs, regressions, etc. as a 
> result of the refactor. Basically we'd just need to test the heck out of it 
> on all platforms, which isn't a bad thing but adds to the LOE for the ANTLR4 
> upgrade, versus a smaller testing "surface" for incremental development of a 
> C/C++ based grammar.
> 
> 
> On May 4, 2017, at 8:51 AM, Andrew Christianson 
>  wrote:
> 
>>> I tried a quick ANTLR4 upgrade myself, it's indeed a big job to refactor 
>>> the existing grammar. Since the source and target for MiNiFi C++ is, well, 
>>> C++, an alternative could be to use lex/yacc (Flex/Bison) [1], Lemon [2], 
>>> Ragel [3], etc. The downside is maintaining two grammars, but we are doing 
>>> that with all the MiNiFi components already. The upside is being able to 
>>> support EL incrementally as the grammar is developed. What do you think?
>> 
>> This seems like a pragmatic approach. What's the level-of-effort required to 
>> do the initial grammar port and set up the build tooling? Less than 
>> refactoring for ANTLR4? I'm not as familiar with the EL grammar situation.
>> 
>> -Andy
>> 
>> From: Matt Burgess 
>> Sent: Thursday, May 4, 2017 8:46:20 AM
>> To: dev@nifi.apache.org
>> Subject: Re: MiNiFi C++ Expression Language
>> 
>> I tried a quick ANTLR4 upgrade myself, it's indeed a big job to refactor the 
>> existing grammar. Since the source and target for MiNiFi C++ is, well, C++, 
>> an alternative could be to use lex/yacc (Flex/Bison) [1], Lemon [2], Ragel 
>> [3], etc. The downside is maintaining two grammars, but we are doing that 
>> with all the MiNiFi components already. The upside is being able to support 
>> EL incrementally as the grammar is developed. What do you think?
>> 
>> Regards,
>> Matt
>> 
>> [1] http://dinosaur.compilertools.net/
>> [2] http://www.hwaci.com/sw/lemon/
>> [3] http://www.colm.net/open-source/ragel/
>> 
>> 
>> 
>> Sent from my iPhone
>>> On May 4, 2017, at 8:13 AM, Marc P.  wrote:
>>> 
>>> Andrew,
>>> I am not aware of it being actively worked [1]. This would require using
>>> ANTLR4, but I don't believe C++ support is well tested [2].  Someone can
>>> correct me if I'm wrong, but there would have to be changes to both sides.
>>> I attempted a quick straw man with grammars, but didn't take it very far
>>> after making initial changes to the grammar. It generated code, but I'm
>>> uncertain of cross platform compatibility with the expression language. If
>>> that's not expected or required that will remove some limitations as a
>>> result of moving to ANTLR4.
>>> 
>>> [1] https://issues.apache.org/jira/browse/MINIFI-140
>>> [2] http://www.soft-gems.net/index.php/tools/49-the-antlr4-c-target-is-here
>>> 
>>> On Thu, May 4, 2017 at 8:07 AM, Andrew Christianson <
>>> andrew.christian...@nextcentury.com> wrote:
>>> 
 All,
 
 I see that we do not have support for the expression language yet in
 MiNiFi C++. Is anyone actively working on this, and if so, is there an ETA?
 If no one is working on it, is there a general plan for how it should be
 implemented? I think I recall seeing references to ANTLR

Re: MiNiFi C++ Expression Language

[Andrew Christianson]

Do I understand correctly that NiFi is currently using ANTLRv3?

From: Matt Burgess 
Sent: Thursday, May 4, 2017 9:05:35 AM
To: dev@nifi.apache.org
Subject: Re: MiNiFi C++ Expression Language

I haven't used Flex/Bison since a trivial example in college, so I'm not sure 
about the LOE for getting that set up, maybe there's a Maven-built project out 
there that we could look at for inspiration, but that seems unlikely :)

An ANTLR4 refactor (assuming the C++ target is in good shape) would give us 
NiFi/MiNiFi EL compatibility (and full-featured EL support in MiNiFi C++), but 
we'd have to accept the risks of introducing bugs, regressions, etc. as a 
result of the refactor. Basically we'd just need to test the heck out of it on 
all platforms, which isn't a bad thing but adds to the LOE for the ANTLR4 
upgrade, versus a smaller testing "surface" for incremental development of a 
C/C++ based grammar.


On May 4, 2017, at 8:51 AM, Andrew Christianson 
 wrote:

>> I tried a quick ANTLR4 upgrade myself, it's indeed a big job to refactor the 
>> existing grammar. Since the source and target for MiNiFi C++ is, well, C++, 
>> an alternative could be to use lex/yacc (Flex/Bison) [1], Lemon [2], Ragel 
>> [3], etc. The downside is maintaining two grammars, but we are doing that 
>> with all the MiNiFi components already. The upside is being able to support 
>> EL incrementally as the grammar is developed. What do you think?
>
> This seems like a pragmatic approach. What's the level-of-effort required to 
> do the initial grammar port and set up the build tooling? Less than 
> refactoring for ANTLR4? I'm not as familiar with the EL grammar situation.
>
> -Andy
> 
> From: Matt Burgess 
> Sent: Thursday, May 4, 2017 8:46:20 AM
> To: dev@nifi.apache.org
> Subject: Re: MiNiFi C++ Expression Language
>
> I tried a quick ANTLR4 upgrade myself, it's indeed a big job to refactor the 
> existing grammar. Since the source and target for MiNiFi C++ is, well, C++, 
> an alternative could be to use lex/yacc (Flex/Bison) [1], Lemon [2], Ragel 
> [3], etc. The downside is maintaining two grammars, but we are doing that 
> with all the MiNiFi components already. The upside is being able to support 
> EL incrementally as the grammar is developed. What do you think?
>
> Regards,
> Matt
>
> [1] http://dinosaur.compilertools.net/
> [2] http://www.hwaci.com/sw/lemon/
> [3] http://www.colm.net/open-source/ragel/
>
>
>
> Sent from my iPhone
>> On May 4, 2017, at 8:13 AM, Marc P.  wrote:
>>
>> Andrew,
>> I am not aware of it being actively worked [1]. This would require using
>> ANTLR4, but I don't believe C++ support is well tested [2].  Someone can
>> correct me if I'm wrong, but there would have to be changes to both sides.
>> I attempted a quick straw man with grammars, but didn't take it very far
>> after making initial changes to the grammar. It generated code, but I'm
>> uncertain of cross platform compatibility with the expression language. If
>> that's not expected or required that will remove some limitations as a
>> result of moving to ANTLR4.
>>
>> [1] https://issues.apache.org/jira/browse/MINIFI-140
>> [2] http://www.soft-gems.net/index.php/tools/49-the-antlr4-c-target-is-here
>>
>> On Thu, May 4, 2017 at 8:07 AM, Andrew Christianson <
>> andrew.christian...@nextcentury.com> wrote:
>>
>>> All,
>>>
>>> I see that we do not have support for the expression language yet in
>>> MiNiFi C++. Is anyone actively working on this, and if so, is there an ETA?
>>> If no one is working on it, is there a general plan for how it should be
>>> implemented? I think I recall seeing references to ANTLR

Re: MiNiFi C++ Expression Language

[Matt Burgess]

I haven't used Flex/Bison since a trivial example in college, so I'm not sure 
about the LOE for getting that set up, maybe there's a Maven-built project out 
there that we could look at for inspiration, but that seems unlikely :) 

An ANTLR4 refactor (assuming the C++ target is in good shape) would give us 
NiFi/MiNiFi EL compatibility (and full-featured EL support in MiNiFi C++), but 
we'd have to accept the risks of introducing bugs, regressions, etc. as a 
result of the refactor. Basically we'd just need to test the heck out of it on 
all platforms, which isn't a bad thing but adds to the LOE for the ANTLR4 
upgrade, versus a smaller testing "surface" for incremental development of a 
C/C++ based grammar.


On May 4, 2017, at 8:51 AM, Andrew Christianson 
 wrote:

>> I tried a quick ANTLR4 upgrade myself, it's indeed a big job to refactor the 
>> existing grammar. Since the source and target for MiNiFi C++ is, well, C++, 
>> an alternative could be to use lex/yacc (Flex/Bison) [1], Lemon [2], Ragel 
>> [3], etc. The downside is maintaining two grammars, but we are doing that 
>> with all the MiNiFi components already. The upside is being able to support 
>> EL incrementally as the grammar is developed. What do you think?
> 
> This seems like a pragmatic approach. What's the level-of-effort required to 
> do the initial grammar port and set up the build tooling? Less than 
> refactoring for ANTLR4? I'm not as familiar with the EL grammar situation.
> 
> -Andy
> 
> From: Matt Burgess 
> Sent: Thursday, May 4, 2017 8:46:20 AM
> To: dev@nifi.apache.org
> Subject: Re: MiNiFi C++ Expression Language
> 
> I tried a quick ANTLR4 upgrade myself, it's indeed a big job to refactor the 
> existing grammar. Since the source and target for MiNiFi C++ is, well, C++, 
> an alternative could be to use lex/yacc (Flex/Bison) [1], Lemon [2], Ragel 
> [3], etc. The downside is maintaining two grammars, but we are doing that 
> with all the MiNiFi components already. The upside is being able to support 
> EL incrementally as the grammar is developed. What do you think?
> 
> Regards,
> Matt
> 
> [1] http://dinosaur.compilertools.net/
> [2] http://www.hwaci.com/sw/lemon/
> [3] http://www.colm.net/open-source/ragel/
> 
> 
> 
> Sent from my iPhone
>> On May 4, 2017, at 8:13 AM, Marc P.  wrote:
>> 
>> Andrew,
>> I am not aware of it being actively worked [1]. This would require using
>> ANTLR4, but I don't believe C++ support is well tested [2].  Someone can
>> correct me if I'm wrong, but there would have to be changes to both sides.
>> I attempted a quick straw man with grammars, but didn't take it very far
>> after making initial changes to the grammar. It generated code, but I'm
>> uncertain of cross platform compatibility with the expression language. If
>> that's not expected or required that will remove some limitations as a
>> result of moving to ANTLR4.
>> 
>> [1] https://issues.apache.org/jira/browse/MINIFI-140
>> [2] http://www.soft-gems.net/index.php/tools/49-the-antlr4-c-target-is-here
>> 
>> On Thu, May 4, 2017 at 8:07 AM, Andrew Christianson <
>> andrew.christian...@nextcentury.com> wrote:
>> 
>>> All,
>>> 
>>> I see that we do not have support for the expression language yet in
>>> MiNiFi C++. Is anyone actively working on this, and if so, is there an ETA?
>>> If no one is working on it, is there a general plan for how it should be
>>> implemented? I think I recall seeing references to ANTLR

Re: MiNiFi C++ Expression Language

[Andrew Christianson]

> I tried a quick ANTLR4 upgrade myself, it's indeed a big job to refactor the 
> existing grammar. Since the source and target for MiNiFi C++ is, well, C++, 
> an alternative could be to use lex/yacc (Flex/Bison) [1], Lemon [2], Ragel 
> [3], etc. The downside is maintaining two grammars, but we are doing that 
> with all the MiNiFi components already. The upside is being able to support 
> EL incrementally as the grammar is developed. What do you think?

This seems like a pragmatic approach. What's the level-of-effort required to do 
the initial grammar port and set up the build tooling? Less than refactoring 
for ANTLR4? I'm not as familiar with the EL grammar situation.

-Andy

From: Matt Burgess 
Sent: Thursday, May 4, 2017 8:46:20 AM
To: dev@nifi.apache.org
Subject: Re: MiNiFi C++ Expression Language

I tried a quick ANTLR4 upgrade myself, it's indeed a big job to refactor the 
existing grammar. Since the source and target for MiNiFi C++ is, well, C++, an 
alternative could be to use lex/yacc (Flex/Bison) [1], Lemon [2], Ragel [3], 
etc. The downside is maintaining two grammars, but we are doing that with all 
the MiNiFi components already. The upside is being able to support EL 
incrementally as the grammar is developed. What do you think?

Regards,
Matt

[1] http://dinosaur.compilertools.net/
[2] http://www.hwaci.com/sw/lemon/
[3] http://www.colm.net/open-source/ragel/



Sent from my iPhone
> On May 4, 2017, at 8:13 AM, Marc P.  wrote:
>
> Andrew,
>  I am not aware of it being actively worked [1]. This would require using
> ANTLR4, but I don't believe C++ support is well tested [2].  Someone can
> correct me if I'm wrong, but there would have to be changes to both sides.
> I attempted a quick straw man with grammars, but didn't take it very far
> after making initial changes to the grammar. It generated code, but I'm
> uncertain of cross platform compatibility with the expression language. If
> that's not expected or required that will remove some limitations as a
> result of moving to ANTLR4.
>
> [1] https://issues.apache.org/jira/browse/MINIFI-140
> [2] http://www.soft-gems.net/index.php/tools/49-the-antlr4-c-target-is-here
>
> On Thu, May 4, 2017 at 8:07 AM, Andrew Christianson <
> andrew.christian...@nextcentury.com> wrote:
>
>> All,
>>
>> I see that we do not have support for the expression language yet in
>> MiNiFi C++. Is anyone actively working on this, and if so, is there an ETA?
>> If no one is working on it, is there a general plan for how it should be
>> implemented? I think I recall seeing references to ANTLR

Re: MiNiFi C++ Expression Language

[Andrew Christianson]

> I am not aware of it being actively worked [1]. This would require using
> ANTLR4, but I don't believe C++ support is well tested [2].  Someone can
> correct me if I'm wrong, but there would have to be changes to both sides.

Both sides as in ANTLR4 and MiNiFi, or as in MiNiFi and NiFi?

> It generated code, but I'm
> uncertain of cross platform compatibility with the expression language. If
> that's not expected or required that will remove some limitations as a
> result of moving to ANTLR4.

Similarly, do you mean compatibility between NiFi and MiNiFi, or the OS/CPU 
platform? If we're talking compatibility between NiFi and MiNiFi on the expr 
language, maybe Aldrin or Joe has the answer on whether this is a design 
goal/requirement.

On a side note, have we looked into porting the grammar to bison? Bison is 
significantly more tested/supported. It is GNU, but my understanding is that 
the generated code is not tied to GNU, just the generator (bison). Calling it 
as part of the build process should be compatible with Apache.

- Andy

From: Marc P. 
Sent: Thursday, May 4, 2017 8:13:15 AM
To: dev@nifi.apache.org
Subject: Re: MiNiFi C++ Expression Language

Andrew,
  I am not aware of it being actively worked [1]. This would require using
ANTLR4, but I don't believe C++ support is well tested [2].  Someone can
correct me if I'm wrong, but there would have to be changes to both sides.
I attempted a quick straw man with grammars, but didn't take it very far
after making initial changes to the grammar. It generated code, but I'm
uncertain of cross platform compatibility with the expression language. If
that's not expected or required that will remove some limitations as a
result of moving to ANTLR4.

[1] https://issues.apache.org/jira/browse/MINIFI-140
[2] http://www.soft-gems.net/index.php/tools/49-the-antlr4-c-target-is-here

On Thu, May 4, 2017 at 8:07 AM, Andrew Christianson <
andrew.christian...@nextcentury.com> wrote:

> All,
>
> I see that we do not have support for the expression language yet in
> MiNiFi C++. Is anyone actively working on this, and if so, is there an ETA?
> If no one is working on it, is there a general plan for how it should be
> implemented? I think I recall seeing references to ANTLR

Re: MiNiFi C++ Expression Language

[Matt Burgess]

I tried a quick ANTLR4 upgrade myself, it's indeed a big job to refactor the 
existing grammar. Since the source and target for MiNiFi C++ is, well, C++, an 
alternative could be to use lex/yacc (Flex/Bison) [1], Lemon [2], Ragel [3], 
etc. The downside is maintaining two grammars, but we are doing that with all 
the MiNiFi components already. The upside is being able to support EL 
incrementally as the grammar is developed. What do you think?

Regards,
Matt

[1] http://dinosaur.compilertools.net/
[2] http://www.hwaci.com/sw/lemon/
[3] http://www.colm.net/open-source/ragel/



Sent from my iPhone
> On May 4, 2017, at 8:13 AM, Marc P.  wrote:
> 
> Andrew,
>  I am not aware of it being actively worked [1]. This would require using
> ANTLR4, but I don't believe C++ support is well tested [2].  Someone can
> correct me if I'm wrong, but there would have to be changes to both sides.
> I attempted a quick straw man with grammars, but didn't take it very far
> after making initial changes to the grammar. It generated code, but I'm
> uncertain of cross platform compatibility with the expression language. If
> that's not expected or required that will remove some limitations as a
> result of moving to ANTLR4.
> 
> [1] https://issues.apache.org/jira/browse/MINIFI-140
> [2] http://www.soft-gems.net/index.php/tools/49-the-antlr4-c-target-is-here
> 
> On Thu, May 4, 2017 at 8:07 AM, Andrew Christianson <
> andrew.christian...@nextcentury.com> wrote:
> 
>> All,
>> 
>> I see that we do not have support for the expression language yet in
>> MiNiFi C++. Is anyone actively working on this, and if so, is there an ETA?
>> If no one is working on it, is there a general plan for how it should be
>> implemented? I think I recall seeing references to ANTLR

Re: MiNiFi C++ Expression Language

[Marc P.]

Andrew,
  I am not aware of it being actively worked [1]. This would require using
ANTLR4, but I don't believe C++ support is well tested [2].  Someone can
correct me if I'm wrong, but there would have to be changes to both sides.
I attempted a quick straw man with grammars, but didn't take it very far
after making initial changes to the grammar. It generated code, but I'm
uncertain of cross platform compatibility with the expression language. If
that's not expected or required that will remove some limitations as a
result of moving to ANTLR4.

[1] https://issues.apache.org/jira/browse/MINIFI-140
[2] http://www.soft-gems.net/index.php/tools/49-the-antlr4-c-target-is-here

On Thu, May 4, 2017 at 8:07 AM, Andrew Christianson <
andrew.christian...@nextcentury.com> wrote:

> All,
>
> I see that we do not have support for the expression language yet in
> MiNiFi C++. Is anyone actively working on this, and if so, is there an ETA?
> If no one is working on it, is there a general plan for how it should be
> implemented? I think I recall seeing references to ANTLR

MiNiFi C++ Expression Language

[Andrew Christianson]

All,

I see that we do not have support for the expression language yet in MiNiFi 
C++. Is anyone actively working on this, and if so, is there an ETA? If no one 
is working on it, is there a general plan for how it should be implemented? I 
think I recall seeing references to ANTLR 

Re: whats new in nifi 1.2.0 / careful use of marks and being clear about non released features

[Anshuman Ghosh]

Hello all,

Thank you for the response!
We have built the NiFi from the master branch on git and able to use the
Google Connectors properly.
Should there be any challenges or observations, will update here.

​Cheers!​

​
__

*Kind Regards,*
*Anshuman Ghosh*
*Contact - +49 179 9090964*


On Fri, Apr 28, 2017 at 4:48 PM, Koji Kawamura 
wrote:

> Hi all,
>
> I should have written about the state of the project clearer, and be
> careful about talking these things.
> I am sorry that I spread it out without proper naming and disclaimer
> in the presentation.
>
> The presentation has been updated with a disclaimer page and title
> page now has "NOT RELEASED YET" note.
>
> The new features list are created originally for myself to catch-up
> what has been added (BUT SUBJECT TO CHANGE), I thought it would be
> helpful for other people, too.
>
> To come up with the contents, I searched JIRA with 1.2.0 and "new
> feature/improvement" and I wrote things only 'Resolved' state.
> To list up new Processors, I used following git command:
>
> git diff rel/nifi-1.1.2 --name-status |grep '^A' |grep java |grep -v
> test > /tmp/added.txt
>
> and
>
> git diff rel/nifi-1.1.2 --name-status  |grep
> META-INF/services/org.apache.nifi.processor.Processor
>
> Then I looked through added source file names to pick up processors.
>
> Joe, thank you for guiding me to the right direction. I hope it will
> not affect our release negatively.
> Looking forward these great new features are come true soon.
>
> Sincerely,
> Koji
>
> On Fri, Apr 28, 2017 at 11:28 PM, Anshuman Ghosh
>  wrote:
> > Thank you Bryan, it will be very helpful :-)
> > Have a great weekend!
> >
> >
> > __
> >
> > *Kind Regards,*
> > *Anshuman Ghosh*
> > *Contact - +49 179 9090964*
> >
> >
> > On Fri, Apr 28, 2017 at 4:15 PM, Bryan Bende  wrote:
> >
> >> Anshuman,
> >>
> >> You shouldn't have to download anything to use the NiFI NAR Maven
> Plugin.
> >>
> >> If you are building the latest NiFi code on the master branch then it
> >> will already be using version 1.2.0:
> >>
> >> https://github.com/apache/nifi/blob/master/pom.xml#L1780
> >>
> >> If you are building your own NARs then you just need to update the
> >> version wherever you declare the plugin.
> >>
> >> In either case, Maven will fetch the plugin from Maven central.
> >>
> >> Thanks,
> >>
> >> Bryan
> >>
> >>
> >> On Fri, Apr 28, 2017 at 10:06 AM, Anshuman Ghosh
> >>  wrote:
> >> > Hello Andre,
> >> >
> >> > Thank you so much for the reply!
> >> > Yes we are excited about the Schema Registry also :-)
> >> > As per the release note "NiFi NAR Maven Plugin Version 1.2.0" was
> >> supposed
> >> > to release by March 17, 2017
> >> > I do not find that here though - https://nifi.apache.org/
> download.html
> >> >
> >> > Any idea how to merge this.
> >> > Thanking you in advance!
> >> >
> >> >
> >> > __
> >> >
> >> > *Kind Regards,*
> >> > *Anshuman Ghosh*
> >> > *Contact - +49 179 9090964*
> >> >
> >> >
> >> > On Fri, Apr 28, 2017 at 3:19 PM, Andre  wrote:
> >> >
> >> >> Anshuman,
> >> >>
> >> >> The GCS processors have been merged to master a few weeks ago so in
> my
> >> >> personal opinion it should be a fairly safe bet to assume they will
> >> make to
> >> >> 1.2.0. Emphasis on assume. :-)
> >> >>
> >> >> From what I gather the code base is reaching stability and as per
> email
> >> >> from Bryan earlier this week we should soon see the RC leaving the
> oven.
> >> >>
> >> >> Not sure if looked at the merge history lately and I certainly don't
> >> want
> >> >> to set high expectations But the changes since 1.1.2 are looking
> >> >> amazing... Like... Truly amazing.
> >> >>
> >> >> But as I said, don't want to set high expectations so hold tight as
> >> Bryan
> >> >> continues to roast the RC.
> >> >>
> >> >> Cheers
> >> >>
> >> >>
> >> >>
> >> >> On Fri, Apr 28, 2017 at 11:01 PM, Anshuman Ghosh <
> >> >> anshuman.ghosh2...@gmail.com> wrote:
> >> >>
> >> >> > Hello Joe,
> >> >> >
> >> >> > I was about to write an email to the community and just then I have
> >> >> > received this email. Thank you!
> >> >> >
> >> >> > I would like to know whether GCS processors are available or not?
> >> >> > We have a requirement to use them.
> >> >> > From where I get this latest version as the one we have downloaded
> >> >> doesn't
> >> >> > seem to have GCS processors.
> >> >> >
> >> >> > Thanking you in advance!
> >> >> >
> >> >> >
> >> >> > __
> >> >> >
> >> >> > *Kind Regards,*
> >> >> > *Anshuman Ghosh*
> >> >> > *Contact - +49 179 9090964*
> >> >> >
> >> >> >
> >> >> > On Fri, Apr 28, 2017 at 2:45 PM, Joe Witt 
> wrote:
> >> >> >
> >> >> > > Team,
> >> >> > >
> >> >> > > Just noticed these slides
> >> >> > > https://www.slideshare.net/KojiKawamura/whats-newnifi120
> >> >> > >
> >> >> > > While the