from:"Andrew Christianson"

Re: MiNiFi C++ Unit Test Example

2017-06-13 Thread Andrew Christianson

Thanks, Marc.

I checked out the TEST_CASE and it looks like an improvement. From my 
perspective on the test harness, the most important thing would be minimizing 
boilerplate increasing the expressiveness of the TEST case. The new test 
harness is a nice step in that direction as it encapsulates a lot of the 
underlying infrastructure. The line count is significantly lower and the lines 
that do exist pertain mostly to what's actually being tested.

I will rebase onto this once merged and use the new plan, unless you had any 
significant changes planned for the near term.

Regards,

Andy

From: Marc <phroc...@apache.org>
Sent: Tuesday, June 13, 2017 12:51:39 PM
To: dev@nifi.apache.org
Subject: Re: MiNiFi C++ Unit Test Example

Hi Andrew,
  I've created a ticket [1] that captures a simple discussion for a test
harness along with a reference to a commit that contains an example. This
example isn't intended to be a complete. I'm sure it will evolve or change
completely, but I wanted to provide an example from which we could work so
we had something to discuss in the ticket. Please let me know your
wants/desires for this harness. Would love some feedback/help on this from
you or anyone else. We can also incorporate gmock to make some of this
easier and am completely open to that idea.

[1] https://issues.apache.org/jira/browse/MINIFI-335

   Best Regards,
   Marc Parisi


On Mon, Jun 5, 2017 at 11:22 AM, Andrew Christianson <
andrew.christian...@nextcentury.com> wrote:

> Sounds good. I will keep an eye out for those merges.
>
> " For example, it would be nice to eat our own dog food by creating
> interfaces for easily instantiating and connecting processors."
>
> Hits the nail on the head. Creating all the underlying
> infrastructure/connections when I really just want to test onTrigger is the
> real headache at this point.
>
> Regards,
>
> Andy
> 
> From: Marc <phroc...@apache.org>
> Sent: Monday, June 5, 2017 10:42:18 AM
> To: dev@nifi.apache.org
> Subject: Re: MiNiFi C++ Unit Test Example
>
> Andrew,
>To come back around to this I wanted to let you know I have a PR to
> complete some of the activities regarding volatile repos. Provenance and
> FlowFile volatile repos are mostly complete with a pending PR to tie in the
> reporting functionality to it. I'm nearly read to update said PR for the
> content repo. The goal here is to move most tests over to volatile repos
> unless those tests specifically need, which I'm testing now. Currently we
> write content to the test folder, I want this to go away before I jump into
> a test harness.
>   In regards to your question about an example minimal test. We have the
> ProcessorTests.cpp and various other processor specific tests; however,
> there is still a lot to add. I'd like to get to a point that we can more
> easily build some of these tests using the code that exists to run the
> agent. For example, it would be nice to eat our own dog food by creating
> interfaces for easily instantiating and connecting processors. I have some
> ideas, so I'll put them up in a ticket once my current effort is finished.
>
> On Tue, May 30, 2017 at 11:19 AM, Marc <phroc...@apache.org> wrote:
>
> > Andrew,
> >That is something we would love to see I think. We could certainly
> > facilitate testing and is something that has been discussed. I'm unsure
> if
> > a ticket exists, though. Feel free to make a ticket to begin building
> some
> > of the test framework for functional components. I'll be happy to help
> when
> > I have some cycles.
> >
> > On May 30, 2017 11:14 AM, "Andrew Christianson" <andrew.christianson@
> > nextcentury.com> wrote:
> >
> >> Hi All,
> >>
> >> I am currently working on final preparations for the merge request on
> >> MINIFI-244. I see that a unit test framework has been added to the code
> >> base. Looking through the extant unit tests, there is a significant
> amount
> >> of boilerplate and it is unclear to me what would be an idiomatic unit
> test
> >> for MiNiFi C++.
> >>
> >> Is there/could someone produce an example minimal unit test source code
> >> for a MiNiFi processor? Alternatively, are there plans/efforts to
> extract
> >> out some of this boilerplate?
> >>
> >> Regards,
> >>
> >> Andy
> >
> >
>

Re: MiNiFi C++ Unit Test Example

2017-06-05 Thread Andrew Christianson

Sounds good. I will keep an eye out for those merges.

" For example, it would be nice to eat our own dog food by creating
interfaces for easily instantiating and connecting processors."

Hits the nail on the head. Creating all the underlying 
infrastructure/connections when I really just want to test onTrigger is the 
real headache at this point.

Regards,

Andy

From: Marc <phroc...@apache.org>
Sent: Monday, June 5, 2017 10:42:18 AM
To: dev@nifi.apache.org
Subject: Re: MiNiFi C++ Unit Test Example

Andrew,
   To come back around to this I wanted to let you know I have a PR to
complete some of the activities regarding volatile repos. Provenance and
FlowFile volatile repos are mostly complete with a pending PR to tie in the
reporting functionality to it. I'm nearly read to update said PR for the
content repo. The goal here is to move most tests over to volatile repos
unless those tests specifically need, which I'm testing now. Currently we
write content to the test folder, I want this to go away before I jump into
a test harness.
  In regards to your question about an example minimal test. We have the
ProcessorTests.cpp and various other processor specific tests; however,
there is still a lot to add. I'd like to get to a point that we can more
easily build some of these tests using the code that exists to run the
agent. For example, it would be nice to eat our own dog food by creating
interfaces for easily instantiating and connecting processors. I have some
ideas, so I'll put them up in a ticket once my current effort is finished.

On Tue, May 30, 2017 at 11:19 AM, Marc <phroc...@apache.org> wrote:

> Andrew,
>That is something we would love to see I think. We could certainly
> facilitate testing and is something that has been discussed. I'm unsure if
> a ticket exists, though. Feel free to make a ticket to begin building some
> of the test framework for functional components. I'll be happy to help when
> I have some cycles.
>
> On May 30, 2017 11:14 AM, "Andrew Christianson" <andrew.christianson@
> nextcentury.com> wrote:
>
>> Hi All,
>>
>> I am currently working on final preparations for the merge request on
>> MINIFI-244. I see that a unit test framework has been added to the code
>> base. Looking through the extant unit tests, there is a significant amount
>> of boilerplate and it is unclear to me what would be an idiomatic unit test
>> for MiNiFi C++.
>>
>> Is there/could someone produce an example minimal unit test source code
>> for a MiNiFi processor? Alternatively, are there plans/efforts to extract
>> out some of this boilerplate?
>>
>> Regards,
>>
>> Andy
>
>

MiNiFi C++ Unit Test Example

2017-05-30 Thread Andrew Christianson

Hi All,

I am currently working on final preparations for the merge request on 
MINIFI-244. I see that a unit test framework has been added to the code base. 
Looking through the extant unit tests, there is a significant amount of 
boilerplate and it is unclear to me what would be an idiomatic unit test for 
MiNiFi C++.

Is there/could someone produce an example minimal unit test source code for a 
MiNiFi processor? Alternatively, are there plans/efforts to extract out some of 
this boilerplate?

Regards,

Andy

Re: MiNiFi C++ Expression Language

2017-05-04 Thread Andrew Christianson

The island grammar sounds very appealing for an MVP. Simple to implement, yet 
it covers a very common EL use case (dynamically inserting attr vals into 
property values). If we have general consensus I would love to see a MINIFI 
JIRA ticket added for this.

-Andy

From: Matt Burgess <mattyb...@gmail.com>
Sent: Thursday, May 4, 2017 11:09:46 AM
To: dev@nifi.apache.org
Subject: Re: MiNiFi C++ Expression Language

What you have sounds good to me. IMO minimum viable product would be an island 
grammar (meaning you can have any characters outside a ${} expression) and 
inside would support an attribute name. Next steps could be nested expressions 
and/or support for functions, added piecemeal as the contributor sees fit. I 
wasn't involved in the early development of the EL grammar, so maybe someone 
who was has some thoughts on a natural evolution.

Regards,
Matt

> On May 4, 2017, at 11:00 AM, Andrew Christianson 
> <andrew.christian...@nextcentury.com> wrote:
>
> My bad, what does the sketch of the plan *look like*?
>
> -Andy
> ____
> From: Andrew Christianson
> Sent: Thursday, May 4, 2017 10:59:07 AM
> To: dev@nifi.apache.org
> Subject: Re: MiNiFi C++ Expression Language
>
> What does the sketch of the plan to do the separate implementation? Write a 
> flex/bison grammar, hook it into the cmake build, and start using it? Any 
> constraints on features or syntax that this separate implementation must 
> support?
>
> -Andy
> 
> From: Matt Burgess <mattyb...@gmail.com>
> Sent: Thursday, May 4, 2017 9:37:34 AM
> To: dev@nifi.apache.org
> Subject: Re: MiNiFi C++ Expression Language
>
> No plans that I know of. In the meantime, EL support for MiNiFi is kind of 
> held hostage, so maybe the separate implementation is more viable in the 
> nearer term. If/When the ANTLR4 upgrade happens, we could replace whatever 
> exists by then with the cross-platform ANTLR target generation, and test the 
> whole kit and caboodle.
>
> Anyone out there familiar with the aforementioned tools (or willing to 
> hand-roll one)? What do you think about this "EL bootstrapping" approach?
>
> Thanks,
> Matt
>
>> On May 4, 2017, at 9:22 AM, Andrew Christianson 
>> <andrew.christian...@nextcentury.com> wrote:
>>
>> Got it. So the crux of the problem is porting from v3 to v4, plus the added 
>> uncertainty of the C++ v4 target.
>>
>> I'm assuming that NiFi wants to eventually get onto v4 anyway. If that's the 
>> case, then porting to v4 is probably the ticket. Are there any concrete 
>> plans to do so in the NiFi mother project yet?
>>
>> -Andy
>> 
>> From: Matt Burgess <mattyb...@gmail.com>
>> Sent: Thursday, May 4, 2017 9:18:00 AM
>> To: dev@nifi.apache.org
>> Subject: Re: MiNiFi C++ Expression Language
>>
>> Correct, the current NiFi EL grammar is ANTLR3.
>>
>>
>>> On May 4, 2017, at 9:12 AM, Andrew Christianson 
>>> <andrew.christian...@nextcentury.com> wrote:
>>>
>>> Do I understand correctly that NiFi is currently using ANTLRv3?
>>> 
>>> From: Matt Burgess <mattyb...@gmail.com>
>>> Sent: Thursday, May 4, 2017 9:05:35 AM
>>> To: dev@nifi.apache.org
>>> Subject: Re: MiNiFi C++ Expression Language
>>>
>>> I haven't used Flex/Bison since a trivial example in college, so I'm not 
>>> sure about the LOE for getting that set up, maybe there's a Maven-built 
>>> project out there that we could look at for inspiration, but that seems 
>>> unlikely :)
>>>
>>> An ANTLR4 refactor (assuming the C++ target is in good shape) would give us 
>>> NiFi/MiNiFi EL compatibility (and full-featured EL support in MiNiFi C++), 
>>> but we'd have to accept the risks of introducing bugs, regressions, etc. as 
>>> a result of the refactor. Basically we'd just need to test the heck out of 
>>> it on all platforms, which isn't a bad thing but adds to the LOE for the 
>>> ANTLR4 upgrade, versus a smaller testing "surface" for incremental 
>>> development of a C/C++ based grammar.
>>>
>>>
>>> On May 4, 2017, at 8:51 AM, Andrew Christianson 
>>> <andrew.christian...@nextcentury.com> wrote:
>>>
>>>>> I tried a quick ANTLR4 upgrade myself, it's indeed a big job to refactor 
>>>>> the existing grammar. Since the source and target for MiNiFi C++ is, 
>>>>> we

Re: MiNiFi C++ Expression Language

2017-05-04 Thread Andrew Christianson

My bad, what does the sketch of the plan *look like*?

-Andy

From: Andrew Christianson
Sent: Thursday, May 4, 2017 10:59:07 AM
To: dev@nifi.apache.org
Subject: Re: MiNiFi C++ Expression Language

What does the sketch of the plan to do the separate implementation? Write a 
flex/bison grammar, hook it into the cmake build, and start using it? Any 
constraints on features or syntax that this separate implementation must 
support?

-Andy

From: Matt Burgess <mattyb...@gmail.com>
Sent: Thursday, May 4, 2017 9:37:34 AM
To: dev@nifi.apache.org
Subject: Re: MiNiFi C++ Expression Language

No plans that I know of. In the meantime, EL support for MiNiFi is kind of held 
hostage, so maybe the separate implementation is more viable in the nearer 
term. If/When the ANTLR4 upgrade happens, we could replace whatever exists by 
then with the cross-platform ANTLR target generation, and test the whole kit 
and caboodle.

Anyone out there familiar with the aforementioned tools (or willing to 
hand-roll one)? What do you think about this "EL bootstrapping" approach?

Thanks,
Matt

> On May 4, 2017, at 9:22 AM, Andrew Christianson 
> <andrew.christian...@nextcentury.com> wrote:
>
> Got it. So the crux of the problem is porting from v3 to v4, plus the added 
> uncertainty of the C++ v4 target.
>
> I'm assuming that NiFi wants to eventually get onto v4 anyway. If that's the 
> case, then porting to v4 is probably the ticket. Are there any concrete plans 
> to do so in the NiFi mother project yet?
>
> -Andy
> 
> From: Matt Burgess <mattyb...@gmail.com>
> Sent: Thursday, May 4, 2017 9:18:00 AM
> To: dev@nifi.apache.org
> Subject: Re: MiNiFi C++ Expression Language
>
> Correct, the current NiFi EL grammar is ANTLR3.
>
>
>> On May 4, 2017, at 9:12 AM, Andrew Christianson 
>> <andrew.christian...@nextcentury.com> wrote:
>>
>> Do I understand correctly that NiFi is currently using ANTLRv3?
>> 
>> From: Matt Burgess <mattyb...@gmail.com>
>> Sent: Thursday, May 4, 2017 9:05:35 AM
>> To: dev@nifi.apache.org
>> Subject: Re: MiNiFi C++ Expression Language
>>
>> I haven't used Flex/Bison since a trivial example in college, so I'm not 
>> sure about the LOE for getting that set up, maybe there's a Maven-built 
>> project out there that we could look at for inspiration, but that seems 
>> unlikely :)
>>
>> An ANTLR4 refactor (assuming the C++ target is in good shape) would give us 
>> NiFi/MiNiFi EL compatibility (and full-featured EL support in MiNiFi C++), 
>> but we'd have to accept the risks of introducing bugs, regressions, etc. as 
>> a result of the refactor. Basically we'd just need to test the heck out of 
>> it on all platforms, which isn't a bad thing but adds to the LOE for the 
>> ANTLR4 upgrade, versus a smaller testing "surface" for incremental 
>> development of a C/C++ based grammar.
>>
>>
>> On May 4, 2017, at 8:51 AM, Andrew Christianson 
>> <andrew.christian...@nextcentury.com> wrote:
>>
>>>> I tried a quick ANTLR4 upgrade myself, it's indeed a big job to refactor 
>>>> the existing grammar. Since the source and target for MiNiFi C++ is, well, 
>>>> C++, an alternative could be to use lex/yacc (Flex/Bison) [1], Lemon [2], 
>>>> Ragel [3], etc. The downside is maintaining two grammars, but we are doing 
>>>> that with all the MiNiFi components already. The upside is being able to 
>>>> support EL incrementally as the grammar is developed. What do you think?
>>>
>>> This seems like a pragmatic approach. What's the level-of-effort required 
>>> to do the initial grammar port and set up the build tooling? Less than 
>>> refactoring for ANTLR4? I'm not as familiar with the EL grammar situation.
>>>
>>> -Andy
>>> 
>>> From: Matt Burgess <mattyb...@gmail.com>
>>> Sent: Thursday, May 4, 2017 8:46:20 AM
>>> To: dev@nifi.apache.org
>>> Subject: Re: MiNiFi C++ Expression Language
>>>
>>> I tried a quick ANTLR4 upgrade myself, it's indeed a big job to refactor 
>>> the existing grammar. Since the source and target for MiNiFi C++ is, well, 
>>> C++, an alternative could be to use lex/yacc (Flex/Bison) [1], Lemon [2], 
>>> Ragel [3], etc. The downside is maintaining two grammars, but we are doing 
>>> that with all the MiNiFi components already. The upside is being able to 
>>> support EL incrementally as the grammar is dev

Re: MiNiFi C++ Expression Language

2017-05-04 Thread Andrew Christianson

What does the sketch of the plan to do the separate implementation? Write a 
flex/bison grammar, hook it into the cmake build, and start using it? Any 
constraints on features or syntax that this separate implementation must 
support?

-Andy

From: Matt Burgess <mattyb...@gmail.com>
Sent: Thursday, May 4, 2017 9:37:34 AM
To: dev@nifi.apache.org
Subject: Re: MiNiFi C++ Expression Language

No plans that I know of. In the meantime, EL support for MiNiFi is kind of held 
hostage, so maybe the separate implementation is more viable in the nearer 
term. If/When the ANTLR4 upgrade happens, we could replace whatever exists by 
then with the cross-platform ANTLR target generation, and test the whole kit 
and caboodle.

Anyone out there familiar with the aforementioned tools (or willing to 
hand-roll one)? What do you think about this "EL bootstrapping" approach?

Thanks,
Matt

> On May 4, 2017, at 9:22 AM, Andrew Christianson 
> <andrew.christian...@nextcentury.com> wrote:
>
> Got it. So the crux of the problem is porting from v3 to v4, plus the added 
> uncertainty of the C++ v4 target.
>
> I'm assuming that NiFi wants to eventually get onto v4 anyway. If that's the 
> case, then porting to v4 is probably the ticket. Are there any concrete plans 
> to do so in the NiFi mother project yet?
>
> -Andy
> 
> From: Matt Burgess <mattyb...@gmail.com>
> Sent: Thursday, May 4, 2017 9:18:00 AM
> To: dev@nifi.apache.org
> Subject: Re: MiNiFi C++ Expression Language
>
> Correct, the current NiFi EL grammar is ANTLR3.
>
>
>> On May 4, 2017, at 9:12 AM, Andrew Christianson 
>> <andrew.christian...@nextcentury.com> wrote:
>>
>> Do I understand correctly that NiFi is currently using ANTLRv3?
>> 
>> From: Matt Burgess <mattyb...@gmail.com>
>> Sent: Thursday, May 4, 2017 9:05:35 AM
>> To: dev@nifi.apache.org
>> Subject: Re: MiNiFi C++ Expression Language
>>
>> I haven't used Flex/Bison since a trivial example in college, so I'm not 
>> sure about the LOE for getting that set up, maybe there's a Maven-built 
>> project out there that we could look at for inspiration, but that seems 
>> unlikely :)
>>
>> An ANTLR4 refactor (assuming the C++ target is in good shape) would give us 
>> NiFi/MiNiFi EL compatibility (and full-featured EL support in MiNiFi C++), 
>> but we'd have to accept the risks of introducing bugs, regressions, etc. as 
>> a result of the refactor. Basically we'd just need to test the heck out of 
>> it on all platforms, which isn't a bad thing but adds to the LOE for the 
>> ANTLR4 upgrade, versus a smaller testing "surface" for incremental 
>> development of a C/C++ based grammar.
>>
>>
>> On May 4, 2017, at 8:51 AM, Andrew Christianson 
>> <andrew.christian...@nextcentury.com> wrote:
>>
>>>> I tried a quick ANTLR4 upgrade myself, it's indeed a big job to refactor 
>>>> the existing grammar. Since the source and target for MiNiFi C++ is, well, 
>>>> C++, an alternative could be to use lex/yacc (Flex/Bison) [1], Lemon [2], 
>>>> Ragel [3], etc. The downside is maintaining two grammars, but we are doing 
>>>> that with all the MiNiFi components already. The upside is being able to 
>>>> support EL incrementally as the grammar is developed. What do you think?
>>>
>>> This seems like a pragmatic approach. What's the level-of-effort required 
>>> to do the initial grammar port and set up the build tooling? Less than 
>>> refactoring for ANTLR4? I'm not as familiar with the EL grammar situation.
>>>
>>> -Andy
>>> 
>>> From: Matt Burgess <mattyb...@gmail.com>
>>> Sent: Thursday, May 4, 2017 8:46:20 AM
>>> To: dev@nifi.apache.org
>>> Subject: Re: MiNiFi C++ Expression Language
>>>
>>> I tried a quick ANTLR4 upgrade myself, it's indeed a big job to refactor 
>>> the existing grammar. Since the source and target for MiNiFi C++ is, well, 
>>> C++, an alternative could be to use lex/yacc (Flex/Bison) [1], Lemon [2], 
>>> Ragel [3], etc. The downside is maintaining two grammars, but we are doing 
>>> that with all the MiNiFi components already. The upside is being able to 
>>> support EL incrementally as the grammar is developed. What do you think?
>>>
>>> Regards,
>>> Matt
>>>
>>> [1] http://dinosaur.compilertools.net/
>>> [2] http://www.hwaci.com/sw/lemon/
>>> [3] http://www.colm.net/o

Re: MiNiFi C++ Expression Language

2017-05-04 Thread Andrew Christianson

Got it. So the crux of the problem is porting from v3 to v4, plus the added 
uncertainty of the C++ v4 target.

I'm assuming that NiFi wants to eventually get onto v4 anyway. If that's the 
case, then porting to v4 is probably the ticket. Are there any concrete plans 
to do so in the NiFi mother project yet?

-Andy

From: Matt Burgess <mattyb...@gmail.com>
Sent: Thursday, May 4, 2017 9:18:00 AM
To: dev@nifi.apache.org
Subject: Re: MiNiFi C++ Expression Language

Correct, the current NiFi EL grammar is ANTLR3.


> On May 4, 2017, at 9:12 AM, Andrew Christianson 
> <andrew.christian...@nextcentury.com> wrote:
>
> Do I understand correctly that NiFi is currently using ANTLRv3?
> 
> From: Matt Burgess <mattyb...@gmail.com>
> Sent: Thursday, May 4, 2017 9:05:35 AM
> To: dev@nifi.apache.org
> Subject: Re: MiNiFi C++ Expression Language
>
> I haven't used Flex/Bison since a trivial example in college, so I'm not sure 
> about the LOE for getting that set up, maybe there's a Maven-built project 
> out there that we could look at for inspiration, but that seems unlikely :)
>
> An ANTLR4 refactor (assuming the C++ target is in good shape) would give us 
> NiFi/MiNiFi EL compatibility (and full-featured EL support in MiNiFi C++), 
> but we'd have to accept the risks of introducing bugs, regressions, etc. as a 
> result of the refactor. Basically we'd just need to test the heck out of it 
> on all platforms, which isn't a bad thing but adds to the LOE for the ANTLR4 
> upgrade, versus a smaller testing "surface" for incremental development of a 
> C/C++ based grammar.
>
>
> On May 4, 2017, at 8:51 AM, Andrew Christianson 
> <andrew.christian...@nextcentury.com> wrote:
>
>>> I tried a quick ANTLR4 upgrade myself, it's indeed a big job to refactor 
>>> the existing grammar. Since the source and target for MiNiFi C++ is, well, 
>>> C++, an alternative could be to use lex/yacc (Flex/Bison) [1], Lemon [2], 
>>> Ragel [3], etc. The downside is maintaining two grammars, but we are doing 
>>> that with all the MiNiFi components already. The upside is being able to 
>>> support EL incrementally as the grammar is developed. What do you think?
>>
>> This seems like a pragmatic approach. What's the level-of-effort required to 
>> do the initial grammar port and set up the build tooling? Less than 
>> refactoring for ANTLR4? I'm not as familiar with the EL grammar situation.
>>
>> -Andy
>> 
>> From: Matt Burgess <mattyb...@gmail.com>
>> Sent: Thursday, May 4, 2017 8:46:20 AM
>> To: dev@nifi.apache.org
>> Subject: Re: MiNiFi C++ Expression Language
>>
>> I tried a quick ANTLR4 upgrade myself, it's indeed a big job to refactor the 
>> existing grammar. Since the source and target for MiNiFi C++ is, well, C++, 
>> an alternative could be to use lex/yacc (Flex/Bison) [1], Lemon [2], Ragel 
>> [3], etc. The downside is maintaining two grammars, but we are doing that 
>> with all the MiNiFi components already. The upside is being able to support 
>> EL incrementally as the grammar is developed. What do you think?
>>
>> Regards,
>> Matt
>>
>> [1] http://dinosaur.compilertools.net/
>> [2] http://www.hwaci.com/sw/lemon/
>> [3] http://www.colm.net/open-source/ragel/
>>
>>
>>
>> Sent from my iPhone
>>> On May 4, 2017, at 8:13 AM, Marc P. <marc.par...@gmail.com> wrote:
>>>
>>> Andrew,
>>> I am not aware of it being actively worked [1]. This would require using
>>> ANTLR4, but I don't believe C++ support is well tested [2].  Someone can
>>> correct me if I'm wrong, but there would have to be changes to both sides.
>>> I attempted a quick straw man with grammars, but didn't take it very far
>>> after making initial changes to the grammar. It generated code, but I'm
>>> uncertain of cross platform compatibility with the expression language. If
>>> that's not expected or required that will remove some limitations as a
>>> result of moving to ANTLR4.
>>>
>>> [1] https://issues.apache.org/jira/browse/MINIFI-140
>>> [2] http://www.soft-gems.net/index.php/tools/49-the-antlr4-c-target-is-here
>>>
>>> On Thu, May 4, 2017 at 8:07 AM, Andrew Christianson <
>>> andrew.christian...@nextcentury.com> wrote:
>>>
>>>> All,
>>>>
>>>> I see that we do not have support for the expression language yet in
>>>> MiNiFi C++. Is anyone actively working on this, and if so, is there an ETA?
>>>> If no one is working on it, is there a general plan for how it should be
>>>> implemented? I think I recall seeing references to ANTLR

Re: MiNiFi C++ Expression Language

2017-05-04 Thread Andrew Christianson

Do I understand correctly that NiFi is currently using ANTLRv3?

From: Matt Burgess <mattyb...@gmail.com>
Sent: Thursday, May 4, 2017 9:05:35 AM
To: dev@nifi.apache.org
Subject: Re: MiNiFi C++ Expression Language

I haven't used Flex/Bison since a trivial example in college, so I'm not sure 
about the LOE for getting that set up, maybe there's a Maven-built project out 
there that we could look at for inspiration, but that seems unlikely :)

An ANTLR4 refactor (assuming the C++ target is in good shape) would give us 
NiFi/MiNiFi EL compatibility (and full-featured EL support in MiNiFi C++), but 
we'd have to accept the risks of introducing bugs, regressions, etc. as a 
result of the refactor. Basically we'd just need to test the heck out of it on 
all platforms, which isn't a bad thing but adds to the LOE for the ANTLR4 
upgrade, versus a smaller testing "surface" for incremental development of a 
C/C++ based grammar.


On May 4, 2017, at 8:51 AM, Andrew Christianson 
<andrew.christian...@nextcentury.com> wrote:

>> I tried a quick ANTLR4 upgrade myself, it's indeed a big job to refactor the 
>> existing grammar. Since the source and target for MiNiFi C++ is, well, C++, 
>> an alternative could be to use lex/yacc (Flex/Bison) [1], Lemon [2], Ragel 
>> [3], etc. The downside is maintaining two grammars, but we are doing that 
>> with all the MiNiFi components already. The upside is being able to support 
>> EL incrementally as the grammar is developed. What do you think?
>
> This seems like a pragmatic approach. What's the level-of-effort required to 
> do the initial grammar port and set up the build tooling? Less than 
> refactoring for ANTLR4? I'm not as familiar with the EL grammar situation.
>
> -Andy
> 
> From: Matt Burgess <mattyb...@gmail.com>
> Sent: Thursday, May 4, 2017 8:46:20 AM
> To: dev@nifi.apache.org
> Subject: Re: MiNiFi C++ Expression Language
>
> I tried a quick ANTLR4 upgrade myself, it's indeed a big job to refactor the 
> existing grammar. Since the source and target for MiNiFi C++ is, well, C++, 
> an alternative could be to use lex/yacc (Flex/Bison) [1], Lemon [2], Ragel 
> [3], etc. The downside is maintaining two grammars, but we are doing that 
> with all the MiNiFi components already. The upside is being able to support 
> EL incrementally as the grammar is developed. What do you think?
>
> Regards,
> Matt
>
> [1] http://dinosaur.compilertools.net/
> [2] http://www.hwaci.com/sw/lemon/
> [3] http://www.colm.net/open-source/ragel/
>
>
>
> Sent from my iPhone
>> On May 4, 2017, at 8:13 AM, Marc P. <marc.par...@gmail.com> wrote:
>>
>> Andrew,
>> I am not aware of it being actively worked [1]. This would require using
>> ANTLR4, but I don't believe C++ support is well tested [2].  Someone can
>> correct me if I'm wrong, but there would have to be changes to both sides.
>> I attempted a quick straw man with grammars, but didn't take it very far
>> after making initial changes to the grammar. It generated code, but I'm
>> uncertain of cross platform compatibility with the expression language. If
>> that's not expected or required that will remove some limitations as a
>> result of moving to ANTLR4.
>>
>> [1] https://issues.apache.org/jira/browse/MINIFI-140
>> [2] http://www.soft-gems.net/index.php/tools/49-the-antlr4-c-target-is-here
>>
>> On Thu, May 4, 2017 at 8:07 AM, Andrew Christianson <
>> andrew.christian...@nextcentury.com> wrote:
>>
>>> All,
>>>
>>> I see that we do not have support for the expression language yet in
>>> MiNiFi C++. Is anyone actively working on this, and if so, is there an ETA?
>>> If no one is working on it, is there a general plan for how it should be
>>> implemented? I think I recall seeing references to ANTLR

Re: MiNiFi C++ Expression Language

2017-05-04 Thread Andrew Christianson

> I tried a quick ANTLR4 upgrade myself, it's indeed a big job to refactor the 
> existing grammar. Since the source and target for MiNiFi C++ is, well, C++, 
> an alternative could be to use lex/yacc (Flex/Bison) [1], Lemon [2], Ragel 
> [3], etc. The downside is maintaining two grammars, but we are doing that 
> with all the MiNiFi components already. The upside is being able to support 
> EL incrementally as the grammar is developed. What do you think?

This seems like a pragmatic approach. What's the level-of-effort required to do 
the initial grammar port and set up the build tooling? Less than refactoring 
for ANTLR4? I'm not as familiar with the EL grammar situation.

-Andy

From: Matt Burgess <mattyb...@gmail.com>
Sent: Thursday, May 4, 2017 8:46:20 AM
To: dev@nifi.apache.org
Subject: Re: MiNiFi C++ Expression Language

I tried a quick ANTLR4 upgrade myself, it's indeed a big job to refactor the 
existing grammar. Since the source and target for MiNiFi C++ is, well, C++, an 
alternative could be to use lex/yacc (Flex/Bison) [1], Lemon [2], Ragel [3], 
etc. The downside is maintaining two grammars, but we are doing that with all 
the MiNiFi components already. The upside is being able to support EL 
incrementally as the grammar is developed. What do you think?

Regards,
Matt

[1] http://dinosaur.compilertools.net/
[2] http://www.hwaci.com/sw/lemon/
[3] http://www.colm.net/open-source/ragel/



Sent from my iPhone
> On May 4, 2017, at 8:13 AM, Marc P. <marc.par...@gmail.com> wrote:
>
> Andrew,
>  I am not aware of it being actively worked [1]. This would require using
> ANTLR4, but I don't believe C++ support is well tested [2].  Someone can
> correct me if I'm wrong, but there would have to be changes to both sides.
> I attempted a quick straw man with grammars, but didn't take it very far
> after making initial changes to the grammar. It generated code, but I'm
> uncertain of cross platform compatibility with the expression language. If
> that's not expected or required that will remove some limitations as a
> result of moving to ANTLR4.
>
> [1] https://issues.apache.org/jira/browse/MINIFI-140
> [2] http://www.soft-gems.net/index.php/tools/49-the-antlr4-c-target-is-here
>
> On Thu, May 4, 2017 at 8:07 AM, Andrew Christianson <
> andrew.christian...@nextcentury.com> wrote:
>
>> All,
>>
>> I see that we do not have support for the expression language yet in
>> MiNiFi C++. Is anyone actively working on this, and if so, is there an ETA?
>> If no one is working on it, is there a general plan for how it should be
>> implemented? I think I recall seeing references to ANTLR

Re: MiNiFi C++ Expression Language

2017-05-04 Thread Andrew Christianson

> I am not aware of it being actively worked [1]. This would require using
> ANTLR4, but I don't believe C++ support is well tested [2].  Someone can
> correct me if I'm wrong, but there would have to be changes to both sides.

Both sides as in ANTLR4 and MiNiFi, or as in MiNiFi and NiFi?

> It generated code, but I'm
> uncertain of cross platform compatibility with the expression language. If
> that's not expected or required that will remove some limitations as a
> result of moving to ANTLR4.

Similarly, do you mean compatibility between NiFi and MiNiFi, or the OS/CPU 
platform? If we're talking compatibility between NiFi and MiNiFi on the expr 
language, maybe Aldrin or Joe has the answer on whether this is a design 
goal/requirement.

On a side note, have we looked into porting the grammar to bison? Bison is 
significantly more tested/supported. It is GNU, but my understanding is that 
the generated code is not tied to GNU, just the generator (bison). Calling it 
as part of the build process should be compatible with Apache.

- Andy

From: Marc P. <marc.par...@gmail.com>
Sent: Thursday, May 4, 2017 8:13:15 AM
To: dev@nifi.apache.org
Subject: Re: MiNiFi C++ Expression Language

Andrew,
  I am not aware of it being actively worked [1]. This would require using
ANTLR4, but I don't believe C++ support is well tested [2].  Someone can
correct me if I'm wrong, but there would have to be changes to both sides.
I attempted a quick straw man with grammars, but didn't take it very far
after making initial changes to the grammar. It generated code, but I'm
uncertain of cross platform compatibility with the expression language. If
that's not expected or required that will remove some limitations as a
result of moving to ANTLR4.

[1] https://issues.apache.org/jira/browse/MINIFI-140
[2] http://www.soft-gems.net/index.php/tools/49-the-antlr4-c-target-is-here

On Thu, May 4, 2017 at 8:07 AM, Andrew Christianson <
andrew.christian...@nextcentury.com> wrote:

> All,
>
> I see that we do not have support for the expression language yet in
> MiNiFi C++. Is anyone actively working on this, and if so, is there an ETA?
> If no one is working on it, is there a general plan for how it should be
> implemented? I think I recall seeing references to ANTLR

MiNiFi C++ Expression Language

2017-05-04 Thread Andrew Christianson

All,

I see that we do not have support for the expression language yet in MiNiFi 
C++. Is anyone actively working on this, and if so, is there an ETA? If no one 
is working on it, is there a general plan for how it should be implemented? I 
think I recall seeing references to ANTLR

Re: MiNiFi CPP indentation confusion (Re: MINIFI-244)

2017-05-02 Thread Andrew Christianson

Sounds good, thanks.

From: Marc P. <marc.par...@gmail.com>
Sent: Tuesday, May 2, 2017 11:12:02 AM
To: dev@nifi.apache.org
Subject: Re: MiNiFi CPP indentation confusion (Re: MINIFI-244)

Andy,
  That will be solved with MINIFI-226. You will only need to call
  REGISTER_RESOURCE(classname) or REGISTER_RESOURCE_AS(classname,name) in
your header file and it will automatically be loadable via the name.

  You are correct though, we can alias the namespace ( and we do in
core/Core.h). Feel free to create a processors namespace alias.

On Tue, May 2, 2017 at 10:59 AM, Andrew Christianson <
andrew.christian...@nextcentury.com> wrote:

> OK, I can work with that. Does the style guide allow for cleaning up this
> bit?
>
>   if (name
>   == 
> org::apache::nifi::minifi::processors::GenerateFlowFile::ProcessorName)
> {
> processor = std::make_shared<
> org::apache::nifi::minifi::processors::GenerateFlowFile>(name,
> uuid);
>   } else if (name
>   == org::apache::nifi::minifi::processors::LogAttribute::ProcessorName)
> {
> processor = std::make_shared<
> org::apache::nifi::minifi::processors::LogAttribute>(name, uuid);
>   } else if (name
>   == org::apache::nifi::minifi::processors::GetFile::ProcessorName) {
> processor =
> std::make_shared GetFile>(name,
>
>  uuid);
>   } else if (name
>   == org::apache::nifi::minifi::processors::PutFile::ProcessorName) {
> processor =
> std::make_shared PutFile>(name,
>
>  uuid);
>   } else if (name
>   == org::apache::nifi::minifi::processors::TailFile::ProcessorName) {
> processor =
> std::make_shared TailFile>(name,
>
> uuid);
>   } else if (name
>   == org::apache::nifi::minifi::processors::ListenSyslog::ProcessorName)
> {
> processor = std::make_shared<
> org::apache::nifi::minifi::processors::ListenSyslog>(name, uuid);
>   } else if (name
> == org::apache::nifi::minifi::processors::ListenHTTP::ProcessorName)
> {
>   processor = std::make_shared<
>   org::apache::nifi::minifi::processors::ListenHTTP>(name, uuid);
>   } else if (name
>   == org::apache::nifi::minifi::processors::InvokeHTTP::ProcessorName)
> {
> processor = std::make_shared<
> org::apache::nifi::minifi::processors::InvokeHTTP>(name,
> uuid);
>   } else if (name
>   == org::apache::nifi::minifi::processors::ExecuteProcess::ProcessorName)
> {
> processor = std::make_shared<
> org::apache::nifi::minifi::processors::ExecuteProcess>(name,
> uuid);
>   } else if (name
>   == org::apache::nifi::minifi::processors::AppendHostInfo::ProcessorName)
> {
> processor = std::make_shared<
> org::apache::nifi::minifi::processors::AppendHostInfo>(name,
> uuid);
>   } else {
> logger_->log_error("No Processor defined for %s", name.c_str());
> return nullptr;
>   }
>
> It seems very excessive to continually repeat 
> org::apache::nifi::minifi::processors.
> It looks like the style guide may allow something like this?
>
> namespace processors = org::apache::nifi::minifi::processors;
>
> Is that correct?
>
> -Andy
> 
> From: Kevin Doran <kdoran.apa...@gmail.com>
> Sent: Tuesday, May 2, 2017 10:49:25 AM
> To: dev@nifi.apache.org
> Subject: Re: MiNiFi CPP indentation confusion (Re: MINIFI-244)
>
> Hi Andy,
>
> I’ll let a minify-cpp veteran jump in if I have this wrong, but I believe
> we are following the Google C++ Style Guide [1]. As it is my first time
> following this style, it caused me to do a double take as well :-)
>
> The Google C++ Style Guide does specify entries under public, protected,
> and private sections are indented one space [2]
>
> [1] https://google.github.io/styleguide/cppguide.html
> [2] https://google.github.io/styleguide/cppguide.html#Class_Format
>
> -- Kevin
>
> On 5/2/17, 10:35, "Andrew Christianson" <andrew.christianson@
> nextcentury.com> wrote:
>
> On closer inspection, it looks like just the public/private/etc.
> specifiers are single-space indented, with the rest being two-spaces. Is
> this intentional?
> 
> From: Andrew Christianson
> Sent: Tuesday, May 2, 2017 10:33:27 AM
> To: dev@nifi.apache.org
> Subject: MiNiFi CPP indentation confusion (Re: MINIFI-244)
>
> Working on merging my code for MINIFI-244 with the latest refactoring
> where all of the classes have been put into nested namespaces. Looking at
> sibling files, it looks like several of them are indented with one space.
> This surprises me. What's the preferred indentation/code style? Two spaces?
> This would be a first for me if we're actually going for single-space
> indentation.
>
> Regards,
>
> Andy
>
>
>
>

Re: MiNiFi CPP indentation confusion (Re: MINIFI-244)

2017-05-02 Thread Andrew Christianson

OK, I can work with that. Does the style guide allow for cleaning up this bit?

  if (name
  == 
org::apache::nifi::minifi::processors::GenerateFlowFile::ProcessorName) {
processor = std::make_shared<
org::apache::nifi::minifi::processors::GenerateFlowFile>(name, uuid);
  } else if (name
  == org::apache::nifi::minifi::processors::LogAttribute::ProcessorName) {
processor = std::make_shared<
org::apache::nifi::minifi::processors::LogAttribute>(name, uuid);
  } else if (name
  == org::apache::nifi::minifi::processors::GetFile::ProcessorName) {
processor =
std::make_shared(name,
 uuid);
  } else if (name
  == org::apache::nifi::minifi::processors::PutFile::ProcessorName) {
processor =
std::make_shared(name,
 uuid);
  } else if (name
  == org::apache::nifi::minifi::processors::TailFile::ProcessorName) {
processor =
std::make_shared(name,
  uuid);
  } else if (name
  == org::apache::nifi::minifi::processors::ListenSyslog::ProcessorName) {
processor = std::make_shared<
org::apache::nifi::minifi::processors::ListenSyslog>(name, uuid);
  } else if (name
== org::apache::nifi::minifi::processors::ListenHTTP::ProcessorName) {
  processor = std::make_shared<
  org::apache::nifi::minifi::processors::ListenHTTP>(name, uuid);
  } else if (name
  == org::apache::nifi::minifi::processors::InvokeHTTP::ProcessorName) {
processor = std::make_shared<
org::apache::nifi::minifi::processors::InvokeHTTP>(name, uuid);
  } else if (name
  == org::apache::nifi::minifi::processors::ExecuteProcess::ProcessorName) {
processor = std::make_shared<
org::apache::nifi::minifi::processors::ExecuteProcess>(name, uuid);
  } else if (name
  == org::apache::nifi::minifi::processors::AppendHostInfo::ProcessorName) {
processor = std::make_shared<
org::apache::nifi::minifi::processors::AppendHostInfo>(name, uuid);
  } else {
logger_->log_error("No Processor defined for %s", name.c_str());
return nullptr;
  }

It seems very excessive to continually repeat 
org::apache::nifi::minifi::processors. It looks like the style guide may allow 
something like this?

namespace processors = org::apache::nifi::minifi::processors;

Is that correct?

-Andy

From: Kevin Doran <kdoran.apa...@gmail.com>
Sent: Tuesday, May 2, 2017 10:49:25 AM
To: dev@nifi.apache.org
Subject: Re: MiNiFi CPP indentation confusion (Re: MINIFI-244)

Hi Andy,

I’ll let a minify-cpp veteran jump in if I have this wrong, but I believe we 
are following the Google C++ Style Guide [1]. As it is my first time following 
this style, it caused me to do a double take as well :-)

The Google C++ Style Guide does specify entries under public, protected, and 
private sections are indented one space [2]

[1] https://google.github.io/styleguide/cppguide.html
[2] https://google.github.io/styleguide/cppguide.html#Class_Format

-- Kevin

On 5/2/17, 10:35, "Andrew Christianson" <andrew.christian...@nextcentury.com> 
wrote:

On closer inspection, it looks like just the public/private/etc. specifiers 
are single-space indented, with the rest being two-spaces. Is this intentional?

From: Andrew Christianson
Sent: Tuesday, May 2, 2017 10:33:27 AM
To: dev@nifi.apache.org
Subject: MiNiFi CPP indentation confusion (Re: MINIFI-244)

Working on merging my code for MINIFI-244 with the latest refactoring where 
all of the classes have been put into nested namespaces. Looking at sibling 
files, it looks like several of them are indented with one space. This 
surprises me. What's the preferred indentation/code style? Two spaces? This 
would be a first for me if we're actually going for single-space indentation.

Regards,

Andy

Re: MiNiFi CPP indentation confusion (Re: MINIFI-244)

2017-05-02 Thread Andrew Christianson

On closer inspection, it looks like just the public/private/etc. specifiers are 
single-space indented, with the rest being two-spaces. Is this intentional?

From: Andrew Christianson
Sent: Tuesday, May 2, 2017 10:33:27 AM
To: dev@nifi.apache.org
Subject: MiNiFi CPP indentation confusion (Re: MINIFI-244)

Working on merging my code for MINIFI-244 with the latest refactoring where all 
of the classes have been put into nested namespaces. Looking at sibling files, 
it looks like several of them are indented with one space. This surprises me. 
What's the preferred indentation/code style? Two spaces? This would be a first 
for me if we're actually going for single-space indentation.

Regards,

Andy

MiNiFi CPP indentation confusion (Re: MINIFI-244)

2017-05-02 Thread Andrew Christianson

Working on merging my code for MINIFI-244 with the latest refactoring where all 
of the classes have been put into nested namespaces. Looking at sibling files, 
it looks like several of them are indented with one space. This surprises me. 
What's the preferred indentation/code style? Two spaces? This would be a first 
for me if we're actually going for single-space indentation.

Regards,

Andy

Re: MiNiFi CPP Processor naming -- Sets attribute to value of content

2017-04-28 Thread Andrew Christianson

Unfortunately, it looks like this one is blocked by dynamic properties [1].  In 
the meantime, we will put together a skeleton which simply matches ".*" and 
puts it into an attr specified by a property which we'll remove when dynamic 
properties are supported. As such, that ExtractText will sit on a branch on 
github until dynamic properties are done and this can be fleshed out properly.

[1]: https://issues.apache.org/jira/browse/MINIFI-171
____
From: Andrew Christianson
Sent: Friday, April 28, 2017 9:26:53 AM
To: dev
Subject: Re: MiNiFi CPP Processor naming -- Sets attribute to value of content

Added https://issues.apache.org/jira/browse/MINIFI-280

Thanks again,

Andy
____
From: Andrew Christianson
Sent: Friday, April 28, 2017 9:24:30 AM
To: dev
Subject: Re: MiNiFi CPP Processor naming -- Sets attribute to value of content

Pierre,

Agree 100%; that's why I asked ;)

Will move forward with ExtractText.

Regards,

Andy

From: Pierre Villard <pierre.villard...@gmail.com>
Sent: Friday, April 28, 2017 8:21:24 AM
To: dev
Subject: Re: MiNiFi CPP Processor naming -- Sets attribute to value of content

Hi Andrew,

The existing equivalent as of now in base processors is ExtractText (it
uses regex to extract data from the content and assigned the result to
attributes). This is probably a good idea to follow the same approach if
possible?

Pierre

2017-04-28 14:02 GMT+02:00 Andrew Christianson <
andrew.christian...@nextcentury.com>:

> NiFi/MiNiFi Developers,
>
> I am currently working on a very simple processor which sets the value of
> a configurable attribute to be the value of the content of the flow
> (interpreted as a UTF-8 string).
>
> The most natural name I can think of is ContentToAttribute, but I'm not
> sure that conforms to the naming standard. There is a "To" processor in
> NiFi (AttributesToJSON), but most start with a verb. What would you call
> this new processor?
>
> Regards,
>
> Andy

Re: MiNiFi CPP Processor naming -- Sets attribute to value of content

2017-04-28 Thread Andrew Christianson

Added https://issues.apache.org/jira/browse/MINIFI-280

Thanks again,

Andy

From: Andrew Christianson
Sent: Friday, April 28, 2017 9:24:30 AM
To: dev
Subject: Re: MiNiFi CPP Processor naming -- Sets attribute to value of content

Pierre,

Agree 100%; that's why I asked ;)

Will move forward with ExtractText.

Regards,

Andy

From: Pierre Villard <pierre.villard...@gmail.com>
Sent: Friday, April 28, 2017 8:21:24 AM
To: dev
Subject: Re: MiNiFi CPP Processor naming -- Sets attribute to value of content

Hi Andrew,

The existing equivalent as of now in base processors is ExtractText (it
uses regex to extract data from the content and assigned the result to
attributes). This is probably a good idea to follow the same approach if
possible?

Pierre

2017-04-28 14:02 GMT+02:00 Andrew Christianson <
andrew.christian...@nextcentury.com>:

> NiFi/MiNiFi Developers,
>
> I am currently working on a very simple processor which sets the value of
> a configurable attribute to be the value of the content of the flow
> (interpreted as a UTF-8 string).
>
> The most natural name I can think of is ContentToAttribute, but I'm not
> sure that conforms to the naming standard. There is a "To" processor in
> NiFi (AttributesToJSON), but most start with a verb. What would you call
> this new processor?
>
> Regards,
>
> Andy

Re: MiNiFi CPP Processor naming -- Sets attribute to value of content

2017-04-28 Thread Andrew Christianson

Pierre,

Agree 100%; that's why I asked ;)

Will move forward with ExtractText.

Regards,

Andy

From: Pierre Villard <pierre.villard...@gmail.com>
Sent: Friday, April 28, 2017 8:21:24 AM
To: dev
Subject: Re: MiNiFi CPP Processor naming -- Sets attribute to value of content

Hi Andrew,

The existing equivalent as of now in base processors is ExtractText (it
uses regex to extract data from the content and assigned the result to
attributes). This is probably a good idea to follow the same approach if
possible?

Pierre

2017-04-28 14:02 GMT+02:00 Andrew Christianson <
andrew.christian...@nextcentury.com>:

> NiFi/MiNiFi Developers,
>
> I am currently working on a very simple processor which sets the value of
> a configurable attribute to be the value of the content of the flow
> (interpreted as a UTF-8 string).
>
> The most natural name I can think of is ContentToAttribute, but I'm not
> sure that conforms to the naming standard. There is a "To" processor in
> NiFi (AttributesToJSON), but most start with a verb. What would you call
> this new processor?
>
> Regards,
>
> Andy

MiNiFi CPP Processor naming -- Sets attribute to value of content

2017-04-28 Thread Andrew Christianson

NiFi/MiNiFi Developers,

I am currently working on a very simple processor which sets the value of a 
configurable attribute to be the value of the content of the flow (interpreted 
as a UTF-8 string).

The most natural name I can think of is ContentToAttribute, but I'm not sure 
that conforms to the naming standard. There is a "To" processor in NiFi 
(AttributesToJSON), but most start with a verb. What would you call this new 
processor?

Regards,

Andy

Re: Revisiting MINIFI-175; minifi-cpp Alpine Dockerfile

2017-04-26 Thread Andrew Christianson

It might be feasible to keep the build as-is (ubuntu-based), create a second 
stage based on alpine or busybox, then copy the minifi dist to the runtime. It 
would then be a question of library compatibility. We could either install the 
runtime deps (hopefully they are named and versioned in a compatible way) with 
apk add or copy the *.so from the build environment. At that point, it would be 
a matter of libc compatibility. Either musl glibc runtime works, or we install 
the runtime glibc for alpine [1] or the base image for it [2]. If we went the 
route of just copying out the linked *.so files, then a busybox base would be 
feasible and possibly smaller.

Is anyone aware of any other system deps minifi-cpp might have other than the 
linked shared libraries & glibc?

[1] https://github.com/sgerrand/alpine-pkg-glibc
[2] https://github.com/frol/docker-alpine-glibc

From: Jeremy Dyer <jdy...@gmail.com>
Sent: Wednesday, April 26, 2017 11:36:17 AM
To: dev@nifi.apache.org
Subject: Re: Revisiting MINIFI-175; minifi-cpp Alpine Dockerfile

I'm all about smaller and alpine. To Aldrin's point I think we just wanted 
something out there at first and fully expected to optimize later

Sent from my iPhone

> On Apr 26, 2017, at 11:29 AM, Bryan Rosander <brosan...@apache.org> wrote:
>
> I'm definitely in favor of smaller images, especially given the nature of
> the subproject.  I think the MiNiFi Java image may be able to benefit from
> Alpine as well.
>
> Thanks,
> Bryan
>
>> On Wed, Apr 26, 2017 at 11:09 AM, Aldrin Piri <aldrinp...@gmail.com> wrote:
>>
>> Don't think there would be any objections to going smaller with Alpine and
>> certainly makes a lot of sense, but I believe initial efforts were
>> problematic and there was a general desire to get a foundation going.  The
>> biggest issue was needing to navigate cross compilation of a given host to
>> the Alpine format which was something our processes are not quite up to at
>> this juncture.  The integrated dev setup in the image avoided having to
>> tackle this and allowed building of the image on any system where Docker
>> could run whether natively or via one of the Docker for $OS setups.
>>
>> Hadn't actually been aware of the multi-stage docker builds before this
>> thread, but certainly seems like a great application after looking through
>> the associated PRs and docs and could be a more immediate win for the
>> particular case above.
>>
>> We could and should certainly strive to make components pluggable, in this
>> case logging, to support such cases where things may be problematic given a
>> target environment.  I don't think I'd be in favor of removing spdlog at
>> this point but could apply similar approaches used elsewhere in the
>> codebase to give us a little more configurability to use an alternate
>> implementation.
>>
>> Would be in favor of opening up an issue to optimize our Docker build and
>> certainly target Alpine with some of the new build magic.  By the time we
>> have it implemented, the multi-stage builds should likely be sufficiently
>> available.
>>
>> On Wed, Apr 26, 2017 at 10:19 AM, Andrew Christianson <
>> andrew.christian...@nextcentury.com> wrote:
>>
>>> Have we considered porting the current Dockerfile from Ubuntu to Alpine?
>>> The ubuntu image, especially with all the build-time deps left in the
>> final
>>> image, is quite bloated. This runs against the design intent of minimal
>>> footprint in minifi-cpp. Further, the new multi-stage Docker builds would
>>> allow us to isolate the build environment in its own stage and allow only
>>> runtime deps to exist in the final image. Overall it would be a much
>>> smaller image.
>>>
>>> Reviewing the JIRA activity for MINIFI-175, it seems the decision to not
>>> use alpine was originally due to the desire to encapsulate the build of
>>> minifi inside the build of the docker image, which for some reason wasn't
>>> compatible with using alpine. On initial testing, it seems alpine's musl
>>> libc has issues with spdlog. Since it may involve some effort and
>> decision
>>> making with regard to which dependencies are used, this is a decision
>>> probably best made by the community.
>>>
>>> Thoughts?
>>

Revisiting MINIFI-175; minifi-cpp Alpine Dockerfile

2017-04-26 Thread Andrew Christianson

Have we considered porting the current Dockerfile from Ubuntu to Alpine? The 
ubuntu image, especially with all the build-time deps left in the final image, 
is quite bloated. This runs against the design intent of minimal footprint in 
minifi-cpp. Further, the new multi-stage Docker builds would allow us to 
isolate the build environment in its own stage and allow only runtime deps to 
exist in the final image. Overall it would be a much smaller image.

Reviewing the JIRA activity for MINIFI-175, it seems the decision to not use 
alpine was originally due to the desire to encapsulate the build of minifi 
inside the build of the docker image, which for some reason wasn't compatible 
with using alpine. On initial testing, it seems alpine's musl libc has issues 
with spdlog. Since it may involve some effort and decision making with regard 
to which dependencies are used, this is a decision probably best made by the 
community.

Thoughts?

Re: C++ Style Guide

2017-02-07 Thread Andrew Christianson

+1 I've made a few contributions and noticed the inconsistent style. I had been 
trying to match the surrounding style(s), but a consistent style 
guide/automated style check will be very welcomed.

From: Andre 
Sent: Monday, February 6, 2017 10:51:20 PM
To: dev@nifi.apache.org
Subject: Re: C++ Style Guide

Aldrin,

I am also aware of someone trying to use googletest to streamline the
creation of cpp test coverage. I believe we may see a PR sometime later
this week.

Cheers

On Tue, Feb 7, 2017 at 8:51 AM, Aldrin Piri  wrote:

> Marc,
>
> I think that makes a lot of sense and certainly would be a +1 for the
> effort.
>
> Would it also make sense to roll in the associated linter [1] and make that
> an available target of build process to help identify compliance?
>
> [1] https://github.com/google/styleguide/tree/gh-pages/cpplint
>
> On Mon, Feb 6, 2017 at 2:51 PM, Marc  wrote:
>
> > Hello Everyone,
> >
> >   I'm submitting a few PRs to the MiNiFi CPP project and would like to
> > discuss a style guide for the C++ development. I'm partial to Google's
> > Style guide:
> >
> > https://google.github.io/styleguide/cppguide.html
> >
> >   Are there any objections to this? As I make PRs I'd like to make some
> > changes in favor of this style guide and wanted some support for or
> against
> > this. I don't intend on making wholesale changes for the sake of the
> style
> > guide, but instead will make these changes iteratively with my PRs.
> >
> >   In the future if there is support for it we can create a pre-commit
> hook
> > to 'beautify the code' with a tool like clang-format or astyle.
> >
> >   Thoughts for/against this style guide?
> >
> >   Thanks,
> >   Marc
> >
>

Re: MiNiFi event-based scheduling?

2017-01-11 Thread Andrew Christianson

All,

Please review/merge pull request 
https://github.com/apache/nifi-minifi-cpp/pull/40 which adds event-based 
scheduling.

Regards,

Andy

P.S. Thanks for the info Mark (missed your email earlier). I did notice the 
bored duration seemed to be implemented in minifi-cpp. Thankfully, we now have 
both. This "true" event scheduler based on native C++ threading constructs 
should get us significantly lower latency than even 10ms.

From: Mark Payne <marka...@hotmail.com>
Sent: Friday, January 6, 2017 10:59:01 AM
To: dev@nifi.apache.org
Subject: Re: MiNiFi event-based scheduling?

Andy,

Of note - with NiFi, the way that the Timer-Based Scheduling agent works, if 
you set a schedule of "0 secs"
NiFi will determine whether or not there is any work that can be done by the 
processor. If the processor has
no work to do, it is considered "bored" and isn't scheduled again for 10 
milliseconds (this is configurable in
nifi.properties, using the "nifi.bored.yield.duration" property, of which the 
default is 10 milliseconds). This is
designed to avoid hammering the CPU.

Off the top of my head, I am not sure if MiNiFi supports this feature or not, 
though. Any idea, Aldrin?

Thanks
-Mark

> On Jan 6, 2017, at 10:51 AM, Aldrin Piri <aldrinp...@gmail.com> wrote:
>
> Great!  Thanks for taking the time to do so, much appreciated.
>
> On Fri, Jan 6, 2017 at 10:48 AM, Andrew Christianson <
> andrew.christian...@nextcentury.com> wrote:
>
>> Got it. Added this as MINIFI-182 with a link to the wiki for design ideas.
>>
>> 
>> From: Aldrin Piri <aldrinp...@gmail.com>
>> Sent: Friday, January 6, 2017 10:03:06 AM
>> To: dev
>> Subject: Re: MiNiFi event-based scheduling?
>>
>> Hey Andy,
>>
>> Your inspection would prove correct and it does not appear we have a JIRA
>> as of yet either.  If you would be interested in creating one, please do,
>> otherwise I shall see to it that it gets added.
>>
>> We are still working to get some of the more foundational framework
>> components established but certainly can see how this would be helpful.
>> Definitely should track this so that it isn't lost in the shuffle and the
>> framework can be refactored to support it.
>>
>> Thanks!
>> --aldrin
>>
>> On Fri, Jan 6, 2017 at 9:56 AM, Andrew Christianson <
>> andrew.christian...@nextcentury.com> wrote:
>>
>>> We noticed that MiNiFi currently only supports timer-based scheduling.
>>> Looking at the code, one could set the timer interval to 0, but this
>> would
>>> result in a hot loop burning 100% of a core. As such, there seems to be
>> no
>>> way presently to have flow file records processed as soon as incoming
>>> records populate the queue. Further, it looks like the FlowController
>> code
>>> is hard-coded against the timer-based scheduling agent, and unfortunately
>>> there is no obvious way to plug-in an alternative scheduler.
>>>
>>> Are there any near/mid-term plans to implement event-based scheduling?
>>>
>>> -Andy
>>>
>>

Re: MiNiFi event-based scheduling?

2017-01-06 Thread Andrew Christianson

Got it. Added this as MINIFI-182 with a link to the wiki for design ideas.

From: Aldrin Piri <aldrinp...@gmail.com>
Sent: Friday, January 6, 2017 10:03:06 AM
To: dev
Subject: Re: MiNiFi event-based scheduling?

Hey Andy,

Your inspection would prove correct and it does not appear we have a JIRA
as of yet either.  If you would be interested in creating one, please do,
otherwise I shall see to it that it gets added.

We are still working to get some of the more foundational framework
components established but certainly can see how this would be helpful.
Definitely should track this so that it isn't lost in the shuffle and the
framework can be refactored to support it.

Thanks!
--aldrin

On Fri, Jan 6, 2017 at 9:56 AM, Andrew Christianson <
andrew.christian...@nextcentury.com> wrote:

> We noticed that MiNiFi currently only supports timer-based scheduling.
> Looking at the code, one could set the timer interval to 0, but this would
> result in a hot loop burning 100% of a core. As such, there seems to be no
> way presently to have flow file records processed as soon as incoming
> records populate the queue. Further, it looks like the FlowController code
> is hard-coded against the timer-based scheduling agent, and unfortunately
> there is no obvious way to plug-in an alternative scheduler.
>
> Are there any near/mid-term plans to implement event-based scheduling?
>
> -Andy
>

MiNiFi event-based scheduling?

2017-01-06 Thread Andrew Christianson

We noticed that MiNiFi currently only supports timer-based scheduling. Looking 
at the code, one could set the timer interval to 0, but this would result in a 
hot loop burning 100% of a core. As such, there seems to be no way presently to 
have flow file records processed as soon as incoming records populate the 
queue. Further, it looks like the FlowController code is hard-coded against the 
timer-based scheduling agent, and unfortunately there is no obvious way to 
plug-in an alternative scheduler.

Are there any near/mid-term plans to implement event-based scheduling?

-Andy

Re: NiFi compilation error - master & 1.1.x branch

2016-12-22 Thread Andrew Christianson

> What version of Java are using? Oracle or OpenJDK, 8 or 9, etc.? Also

$ java -version 
   
java version "1.8.0_45" 


Java(TM) SE Runtime Environment (build 1.8.0_45-b14)


Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)

> which branch are you building from? It looks like it's building

Tried both of these:

$ git branch -v 
   
* master 44c9ea0 NIFI-3236 - SplitJson performance improvements 


  support/nifi-1.1.x 986e716 NIFI-3188: Added unit test to test corner cases

NiFi compilation error - master & 1.1.x branch

2016-12-21 Thread Andrew Christianson

Hi All,


Trying to compile NiFi. It keeps failing in the nifi-scripting-processors 
project. Tried repeated clean builds. This is on a Ubuntu 14.04 box. Ideas?


Here's the full build output:


nifi-nar-bundles/nifi-scripting-bundle/nifi-scripting-processors$ mvn install
[INFO] Scanning for projects...
[INFO] Inspecting build with total of 1 modules...
[INFO] Installing Nexus Staging features:
[INFO]   ... total of 1 executions of maven-deploy-plugin replaced with 
nexus-staging-maven-plugin
[INFO]
[INFO] 
[INFO] Building nifi-scripting-processors 1.1.1-SNAPSHOT
[INFO] 
[INFO]
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-maven) @ 
nifi-scripting-processors ---
[INFO]
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ 
nifi-scripting-processors ---
[INFO]
[INFO] --- maven-resources-plugin:2.7:resources (default-resources) @ 
nifi-scripting-processors ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.2:compile (default-compile) @ 
nifi-scripting-processors ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-resources-plugin:2.7:testResources (default-testResources) @ 
nifi-scripting-processors ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 27 resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.2:testCompile (default-testCompile) @ 
nifi-scripting-processors ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-compiler-plugin:3.2:testCompile (groovy-tests) @ 
nifi-scripting-processors ---
[INFO] Changes detected - recompiling the module!
[INFO] Using Groovy-Eclipse compiler to compile both Java and Groovy files
[INFO] Found location 

 for className 
[INFO] no javaAgentClass seems to be set
[INFO] Compiling in a forked process using 
/home/user/.m2/repository/org/codehaus/groovy/groovy-eclipse-batch/2.4.3-01/groovy-eclipse-batch-2.4.3-01.jar
[INFO]
[INFO] --- maven-surefire-plugin:2.18:test (default-test) @ 
nifi-scripting-processors ---
[INFO] Surefire report directory: 
/home/user/nifi/nifi-nar-bundles/nifi-scripting-bundle/nifi-scripting-processors/target/surefire-reports
[INFO] Using configured provider org.apache.maven.surefire.junit4.JUnit4Provider

---
 T E S T S
---
Running org.apache.nifi.processors.script.TestExecuteJavascript
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.142 sec - in 
org.apache.nifi.processors.script.TestExecuteJavascript
Running org.apache.nifi.processors.script.TestInvokeJython
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.211 sec - in 
org.apache.nifi.processors.script.TestInvokeJython
Running org.apache.nifi.processors.script.TestExecuteJython
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.034 sec - in 
org.apache.nifi.processors.script.TestExecuteJython
Running org.apache.nifi.processors.script.TestExecuteGroovy
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.279 sec - in 
org.apache.nifi.processors.script.TestExecuteGroovy
Running org.apache.nifi.processors.script.TestExecuteLua
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.052 sec - in 
org.apache.nifi.processors.script.TestExecuteLua
Running org.apache.nifi.processors.script.TestInvokeGroovy
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.595 sec - in 
org.apache.nifi.processors.script.TestInvokeGroovy
Running org.apache.nifi.processors.script.TestInvokeJavascript
Tests run: 5, Failures: 3, Errors: 0, Skipped: 0, Time elapsed: 0.505 sec <<< 
FAILURE! - in org.apache.nifi.processors.script.TestInvokeJavascript
testReadFlowFileContentAndStoreInFlowFileAttribute(org.apache.nifi.processors.script.TestInvokeJavascript)
  Time elapsed: 0.206 sec  <<< FAILURE!
java.lang.AssertionError: Processor has 1 validation failures:
'Validation' validated against 
'target/test/resources/javascript/test_reader.js' is invalid because An error 
occurred calling validate in the configured script Processor.

at org.junit.Assert.fail(Assert.java:88)
at 
org.apache.nifi.util.MockProcessContext.assertValid(MockProcessContext.java:251)
at 
org.apache.nifi.util.StandardProcessorTestRunner.assertValid(StandardProcessorTestRunner.java:334)
at 
org.apache.nifi.processors.script.TestInvokeJavascript.testReadFlowFileContentAndStoreInFlowFileAttribute(TestInvokeJavascript.java:59)

testScriptDefinedRelationship(org.apache.nifi.processors.script.TestInvokeJavascript)
  Time elapsed: 0.102 sec  <<< FAILURE!
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at

Re: MiNiFi C++ Unit Test Example

Re: MiNiFi C++ Unit Test Example

MiNiFi C++ Unit Test Example

Re: MiNiFi C++ Expression Language

Re: MiNiFi C++ Expression Language

Re: MiNiFi C++ Expression Language

Re: MiNiFi C++ Expression Language

Re: MiNiFi C++ Expression Language

Re: MiNiFi C++ Expression Language

Re: MiNiFi C++ Expression Language

MiNiFi C++ Expression Language

Re: MiNiFi CPP indentation confusion (Re: MINIFI-244)

Re: MiNiFi CPP indentation confusion (Re: MINIFI-244)

Re: MiNiFi CPP indentation confusion (Re: MINIFI-244)

MiNiFi CPP indentation confusion (Re: MINIFI-244)

Re: MiNiFi CPP Processor naming -- Sets attribute to value of content

Re: MiNiFi CPP Processor naming -- Sets attribute to value of content

Re: MiNiFi CPP Processor naming -- Sets attribute to value of content

MiNiFi CPP Processor naming -- Sets attribute to value of content

Re: Revisiting MINIFI-175; minifi-cpp Alpine Dockerfile

Revisiting MINIFI-175; minifi-cpp Alpine Dockerfile

Re: C++ Style Guide

Re: MiNiFi event-based scheduling?

Re: MiNiFi event-based scheduling?

MiNiFi event-based scheduling?

Re: NiFi compilation error - master & 1.1.x branch

NiFi compilation error - master & 1.1.x branch

27 matches

Site Navigation

Mail list logo

Footer information