Re: Re:[EXT] [DISCUSS] Predictive Analytics for NiFi Metrics

[Andy Christianson]

 > [cid:image003.png@01D55697.965C27D0]
> > > > A similar display for flowfile count is displayed as well.
> > > > The current state of work can be found at
> > > > https://github.com/jmark99/nifi/tree/time-to-overflow
> > > > I welcome your (or any others) feedback on this effort.
> > > > Thanks,
> > > > Mark
> > > > P.S. If the images are not displaying, they can be viewed at
> > > > https://github.com/jmark99/nifi-images
> > > > -Original Message-
> > > > From: Yolanda Davis  > > > yolanda.m.da...@gmail.com>>
> > > > Sent: Monday, August 19, 2019 11:29 AM
> > > > To: dev@nifi.apache.orgmailto:dev@nifi.apache.org
> > > > Subject: Re:[EXT] [DISCUSS] Predictive Analytics for NiFi Metrics
> > > > Hello All,
> > > > I just wanted to follow up on the discussion we started a couple of
> > > > weeks
> > >
> > > > ago concerning an analytics framework for NiFi metrics. Working with
> > > > Andy
> > > > Christianson and Matt Burgess we shaped our ideas and drafted a
> > > > proposal
> > >
> > > > for this feature on the Apache NiFi Wiki [1] . We've also begun
> > > > implementing some of these ideas in a feature branch (which is work in
> > > > progress) [2]. We’d appreciate any questions or feedback you may have.
> > > > Thanks,
> > > > -yolanda
> > > > [1] -
> >
> > https://cwiki.apache.org/confluence/display/NIFI/Operational+Analytics+Framework+for+NiFi
> >
> > > > [2] - https://github.com/apache/nifi/commits/analytics-framework
> > > > On Wed, Jul 31, 2019 at 9:58 AM Andy Christianson <
> > > > aichr...@protonmail.com
> > > > .invalidmailto:aichr...@protonmail.com.invalid> wrote:
> > > >
> > > > > As someone who operated a 24/7 mission-critical NiFi flow, this
> > > >
> > > > > feature would have been a life saver. If I'm heading home on a
> > > > > Friday,
> > >
> > > > > it would be great to have some blinking red lights to let me know
> > > > > that
> > >
> > > > > the system predicts that it is going to experience backpressure
> > > >
> > > > > sometime over the weekend, so that corrective action could be taken
> > > > > before leaving.
> > > >
> > > > >
> > > >
> > > > > Since there is support in the community for this, I created a JIRA to
> > > >
> > > > > track the effort:
> > > >
> > > > >
> > > >
> > > > > https://issues.apache.org/jira/browse/NIFI-6510
> > > >
> > > > >
> > > >
> > > > > I also created a JIRA to track the remote protocol:
> > > >
> > > > >
> > > >
> > > > > https://issues.apache.org/jira/browse/NIFI-6511
> > > >
> > > > >
> > > >
> > > > >
> > > >
> > > > > Regards,
> > > >
> > > > >
> > > >
> > > > > Andy
> > > >
> > > > >
> > > >
> > > > >
> > > >
> > > > > Sent from ProtonMail, Swiss-based encrypted email.
> > > >
> > > > >
> > > >
> > > > > ‐‐‐ Original Message ‐‐‐
> > > >
> > > > > On Wednesday, July 31, 2019 6:57 AM, Arpad Boda  > > > > mailto:ab...@apache.org> wrote:
> > > >
> > > > >
> > > >
> > > > > > If you could share a bit more details about your OPC and Modbus
> > > >
> > > > > > usage,
> > > >
> > > > > that
> > > >
> > > > > > would be highly appreciated!
> > > >
> > > > > >
> > > >
> > > > > > On Wed, Jul 31, 2019 at 12:01 PM Craig Knell craig.kn...@gmail.com
> > > > > > mailto:craig.kn...@gmail.com
> > > >
> > > > > wrote:
> > > >
> > > > > >
> > > >
> > > > > > > Sounds. Great
> > > >
> > > > > > > Let me know if you need some help
> > > >
> > > > > > > Best regards
> > > >
> > > > > > > Craig
> &

Re: [DISCUSS] Predictive Analytics for NiFi Metrics

[Andy Christianson]

As someone who operated a 24/7 mission-critical NiFi flow, this feature would 
have been a life saver. If I'm heading home on a Friday, it would be great to 
have some blinking red lights to let me know that the system predicts that it 
is going to experience backpressure sometime over the weekend, so that 
corrective action could be taken before leaving.

Since there is support in the community for this, I created a JIRA to track the 
effort:

https://issues.apache.org/jira/browse/NIFI-6510

I also created a JIRA to track the remote protocol:

https://issues.apache.org/jira/browse/NIFI-6511


Regards,

Andy


Sent from ProtonMail, Swiss-based encrypted email.

‐‐‐ Original Message ‐‐‐
On Wednesday, July 31, 2019 6:57 AM, Arpad Boda  wrote:

> If you could share a bit more details about your OPC and Modbus usage, that
> would be highly appreciated!
>
> On Wed, Jul 31, 2019 at 12:01 PM Craig Knell craig.kn...@gmail.com wrote:
>
> > Sounds. Great
> > Let me know if you need some help
> > Best regards
> > Craig
> >
> > > On 31 Jul 2019, at 17:31, Arpad Boda ab...@cloudera.com.invalid wrote:
> > > Craig,
> > > OPC ( https://issues.apache.org/jira/browse/MINIFICPP-819 ) and Modbus (
> > > https://issues.apache.org/jira/browse/MINIFICPP-897 ) are on the way for
> > > MiNiFi c++, hopefully both will be part of next release (0.7.0).
> > > It's gonna be legen... wait for it! :)
> > > Regards,
> > > Arpad
> > >
> > > > On Wed, Jul 31, 2019 at 2:30 AM Craig Knell craig.kn...@gmail.com
> > > > wrote:
> > >
> > > > Hi Folks
> > > > That's our use case now. All our Models are run in python.
> > > > Currently we send events to the ML via http, although this is not
> > > > optimal
> > >
> > > > Our use case is edge ML where we want a light weight wrapper for
> > > > Python code base.
> > > > Jython however does not work with the code base
> > > > I'm think of changing the interface to some thing like REDIS for pub/sub
> > > > Id also like this to be a push deployment via minifi
> > > > Also support for sensors via protocols via Modbus and OPC would be great
> > > > Craig
> > > >
> > > > > On Wed, Jul 31, 2019 at 1:43 AM Joe Witt joe.w...@gmail.com wrote:
> > > > > Definitely something that I think would really help the community. It
> > > > > might make sense to frame/structure these APIs such that an internal
> > > > > option
> > > > > could be available to reduce dependencies and get up and running but
> > > > > that
> > >
> > > > > also just as easily a remote implementation where the engine lives and
> > > > > is
> > >
> > > > > managed externally could also be supported.
> > > > > Thanks
> > > > > On Tue, Jul 30, 2019 at 1:40 PM Andy LoPresto alopre...@apache.org
> > > > > wrote:
> > > > >
> > > > > > Yolanda,
> > > > > > I think this sounds like a great idea and will be very useful to
> > > > > > admins/users, as well as enabling some interesting next-level
> > > > > > functionality
> > > > >
> > > > > > and insight generation. Thanks for putting this out there.
> > > > > > Andy LoPresto
> > > > > > alopre...@apache.org
> > > > > > alopresto.apa...@gmail.com
> > > > > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> > > > > >
> > > > > > > On Jul 30, 2019, at 5:55 AM, Yolanda Davis <
> > > > > > > yolanda.m.da...@gmail.com>
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hello Everyone,
> > > > > > > I wanted to reach out to the community to discuss potentially
> > > > > > > enhancing
> > > > >
> > > > > > > NiFi to include predictive analytics that can help users assess 
> > > > > > > and
> > > > > > > predict
> > > > > > > NiFi behavior and performance. Currently NiFi has lots of metrics
> > > > > > > available
> > > > > > > for areas including jvm and flow component usage (via component
> > > > > > > status)
> > > > >
> > > > > > as
> > > > > >
> > > > > > > well as pro

"Node Group" property of FileAccessPolicyProvider

[Andy Christianson]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hi All,

Currently FileAccessPolicyProvider supports specification of a static set of
node identities. This is limiting in environments where the set of node
identities is changing over time, for example during scale-up/down operations
when NiFi is deployed to a clustering environment (e.g. Kubernetes).

I have authored ticket NIFI-5542 [1] proposing a new "Node Group" property. All
users added to this group will be treated as nodes. The group will be populated
by a UserGroupProvider which dynamically provides the set of node identities
that exist in the cluster. The UserGroupProvider will depend on the cluster
environment NiFi is currently deployed to. In the future we may want to
consider offering UserGroupProviders for a set of standard cluster
environments, but that is out of scope for this initial change.

How does the community feel about this proposed change? Is this a good way to
add initial support for authorizing a dynamic set of NiFi nodes in a dynamic
cluster environment?

Regards,

Andy I.C.

1: https://issues.apache.org/jira/browse/NIFI-5542?filter=-2
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.22 (GNU/Linux)

iQEcBAEBAgAGBQJbfB6/AAoJEG1+mBKNMpID1kYH/2Fl6nTnunKkV1L0P1ls/gAZ
Nu4KGS4RB0KZLl910IuYznIaRerQPIfw/bcJUJvcMJUGaSItxqRZkd7XuucjM2dj
MoFIbvoiAGbTfKteF41yuj6iWmDuDGTMFRDf2ZDwuo4bbHdbXIt0IpEAzYW186e0
D+Mzyz53/kkHxyKFFhuIII1hr93yG9leN+E7HTtEeZplpmuXQGXwf9s470TuD9mw
7YVeF9fLt8JB52hZ6E3s9q0wvf2ORkSNAL87YEN++ojPIcQOPyslIsyyu/zwycw5
lWHeDZKh+SvS2IE2jwefSOPRYl6Z9wp0uggRMayiU4+7z5XtlVsdn7TtGYR7nFA=
=NNPm
-END PGP SIGNATURE-

Sent from [ProtonMail](https://protonmail.com), Swiss-based encrypted email.

Re: [DISCUSS] First Release of NiFi Flow Design System

[Andy Christianson]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

> Team,
> 
> There has been a lot of great work done on the NiFi Flow Design System [1]
> and although there are still a few 'in progress' issues ([3], [4], [5]) I 
> think
> we are probably very close to an initial release focused on a reusable
> platform providing a consistent set of UI/UX components across the NiFi
> ecosystem.
> 
> Since NiFi Registry will have a dependency on client code provided by the flow
> design system, the first release of the flow design system would need to
> occur before the next release of NiFi Registry that integrates with it. The
> work on the NiFi Registry side is being done as part of NIFIREG-39 [2].
> 
> Let me know if anyone has any thoughts or comments.  I'm happy to act as RM
> if no one else is interested in doing so, and we can start the process of
> going through JIRA to see what is left.
> 
> Thanks,
> 
> Scott
> 
> [1] https://nifi.apache.org/fds.html
> [2] https://issues.apache.org/jira/browse/NIFIREG-39
> [3] https://issues.apache.org/jira/browse/NIFI-5219
> [4] https://issues.apache.org/jira/browse/NIFI-5216
> [5] https://issues.apache.org/jira/browse/NIFI-5240

Glad to see the progress! FYI, the FDS link [1] appears to be broken (404 Not
Found)

Regards,

Andy I.C.

[1] https://nifi.apache.org/fds.html
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.22 (GNU/Linux)

iQEcBAEBAgAGBQJbCEm2AAoJEG1+mBKNMpIDbhEH/2UdqgAh2F7F8rvIYIqPbX53
Ucjk8SlCLSbmocS8vIx4SbIQamy6bEYgtdqPwUvyfm82QLUW2zDt+1WtHYVMNOlV
dikiNIHRR/yqPCU8ZfNkyKJI0ajw+lvc/DSmH97TYsmmYw889lX/LfnLuObq0RG1
pflvme1E7vDr0CFLSJ39qBnKeTL/AvVDINBXjmdDr/6LlK9Bw/XdHdkMnFZ4dpUF
/mHeNfp+RtHyAHP+Igw1aqaSiNyRWArQlF/T3WHhR9kpyjq+Fca+bl5OMufKxjiq
ATLU6UsoRbBDwmlf/ZOI+O/CKNfD+oHzKFSPpIVsxdkJHWp3huaJiuGyUlbxUhA=
=q33Z
-END PGP SIGNATURE-



​Sent from ProtonMail, Swiss-based encrypted email.​

Re: [DISCUSS] Apache NiFi MiNiFi C++ 0.5.0

[Andy Christianson]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

> Sounds great and I think we've gotten some important changes into the
> codebase from when this first started.  Do we have all the
> appropriate/needed issues tagged for this release?

I support this release.

For this release, I would like to bring special attention to MINIFICPP-281,
which has been marked "RESOLVED." In order for that to work, whoever builds the
.tar.gz should build the binary with all static linking options enabled, which
will minimize dependencies on libraries which may only be on the packager's
system, while maximizing portability of the tarball release.

Regards,

Andy I.C.
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.22 (GNU/Linux)

iQEcBAEBAgAGBQJbBYVvAAoJEG1+mBKNMpIDH9sH/jnbw3hqxBVSZcZwimy2bmEN
P64YR3k6Q+OT2pCIp3e88aAHLLy3KUGlZiltp7viiAiNIcHFAmEkbUfpsDEZB6On
EyZJMV1NhEx3zDgapfhY3BUcarOgOjL77ZcVCuw3OU0LAkZFrXxG7Ok9YTY2smP8
ve4Geck/O8aCtGL5hnNHecJGrm7KsVKbpUeutvwMMTLMNOPwv7ukxLp0o6QufR54
cA0nIPJmLrbuCOWWJzSAzLXHu3Wzox9UQ9htNOHgqmzYCAEpMDZrHGE5xJs9K3Iy
tLVRebJdQfvH+woKWzmwsqrgf3mSoY2iohza/46Tu3+OvETvVtS44RnFz3paKek=
=WN7S
-END PGP SIGNATURE-

​Sent from ProtonMail, Swiss-based encrypted email.​

Re: minificpp AppVeyor

[Andy Christianson]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

> None here.  I think it was initially added by accident when we were
> initializing both MiNiFi Java and C++.  There have been some requests for
> Windows specific functionality that would have that come in handy.  To
> avoid the churn for ASF Infra, we could potentially set it up so that it is
> a no-op should we need it later.  Either way, no strong preferences.

Keeping it online for the eventuality of native Windows builds is a compelling
idea.

In the meantime, how do we make it a noop? I don't see any Appveyor files in
the tree. Will it require an infra ticket to make it a noop?

Regards,

Andy I.C.
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.22 (GNU/Linux)

iQEcBAEBAgAGBQJbBYKWAAoJEG1+mBKNMpIDIdQIAKDfYUhXAAPQdCatVTkc+Y6Z
Wpvt0ldxA0TnpLcIOJURWOf0TPbe+pFvSwaaIQ+Xjb5NvgbPoHnIAlVkIJMORVko
224Xb/YpnncYBGYhNN/tGzIrtcvGVyD+9mEh46O4XKJ5Mq5xWOqlf96pNp6qo62D
4TC+/uyCCi8yVSh1PPzTC7Fi3MnXR4iWQUThnTUrg4XAFrBI6J+ER0Z3KyZrbAX+
ljpZQESWHrm9KE0QU1ihAxzH9vbsVZyZvn6xYp/LS2PAZsXpu3xxGHUtlZVUNmuE
UrxnbGq89pUQKKIPAxmt137O5EzSUOB/SDv2fg/m/KdbTRG7LIwnxqxfo8eN9WE=
=dGSA
-END PGP SIGNATURE-


​Sent from ProtonMail, Swiss-based encrypted email.​

minificpp AppVeyor

[Andy Christianson]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

minificpp devs:

The AppVeyor builds are broken 100% of the time and don't seem to be adding any
value. Are there any objections to the removal of AppVeyor?

Regards,

Andy I.C.
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.22 (GNU/Linux)

iQEcBAEBAgAGBQJbBD+9AAoJEG1+mBKNMpIDo1kIAKhADE74kaQSJKFDW66uQNr8
a7Us6hev4PuRgy79oAyTR8ykMXXmkPAqRskA7bScgP7GV5KWQGLfaQ0cigDsQed1
8DwtfTLVhUutkV5jz7PcQxKgWFXW4dL3oxV8farFfrev5CQ8xNugA+7wRFWI1zxJ
4qC2CzCTd10p3VEBbIhaaGEVg62tSjuXPt81SdJkGfB/rRRzwgYyipfBEEsQLyI5
aN/Qh/r7umeJlZ4un7xDMCTa/eZCHDuGQnpi2HOA3qbEwupXPib22sGKumUnGn5v
jQM/zsCyB52nR6nBFytA6uGwfyrmazXHmDOBVEkd997PaVnUrKFZT6Mnn4uQmTE=
=jj0/
-END PGP SIGNATURE-

Sent from [ProtonMail](https://protonmail.com), Swiss-based encrypted email.

Re: [DISCUSS] MiNiFi Flow Designer

[Andy Christianson]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Great discussion on this topic.

Something I'd like to add which I haven't seen yet, is a MiNiFi production
environment emulator, perhaps talking to a virtualization system like QEMU, to
be used during the flow design & development process. The purpose of this would
be to provide some level of verification for an Edge/IoT flow before it is
deployed out to devices. The minifi binary used in the emulation environment
would hopefully be the same binary in the production environment, so at least
we can verify that the config.yml produced is compatible with the minifi binary
used.

I'm not sure whether this should be integrated into the design/development UI
or whether it should be a separate tool that can be used as part of the process
and potentially integrated with CI/CD systems.

Regards,

Andy I.C.
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.22 (GNU/Linux)

iQEcBAEBAgAGBQJanVOOAAoJEG1+mBKNMpIDV+QIAI3Ii1OQs5D2oOydhcfR765I
cbuRTQZc0uiWhrPrdppyqaUkiSmpuW3Z/mi+vM+tbLIVrheMF0EZ7WQ1DKWKwdZJ
r2I/0CyQ5mkhWRLZDjz4YyKsWXJznrubSz4O2rExjCENiMCsPT/OU3Qpfu7HH9nE
ug0aD17q/KGxLLv8WnT8xlW74PSOvHRdtuGI7kiuZtf9NqSY+V0M8uxp71T9UCuS
j7xDeApCRlYsDuKxc7tg3qdnIROOPhfxKqf7nP6Miymt6cThiGSDSlII01RnkFWP
0VK/S3EWuWAoAe8L31W/Mzdfh+WmzUqLrFsmqmO9yz0Y6nswKecAV8setGL0p9Q=
=BCWa
-END PGP SIGNATURE-

​Sent from ProtonMail, Swiss-based encrypted email.​

Re: [DISCUSS] MiNiFi Flow Designer

[Andy Christianson]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

> I think there are several reasons that this is reasonable. Seeing some of
> the discussion around the consolidation of common UI components presented
> by Scott Aslan, I think sharing the core of the UI elements between the
> traditional NiFi and MiNiFi flow designs is a logical extension.  While
> those flows in MiNiFi are not real-time changes we are dealing with the
> same design elements in creating directed graphs.  Additionally, while
> MiNiFi instances will likely not ever have the same breadth of extensions
> supported, there is nothing inherently prohibitive of a more complex graph
> comprised of larger numbers of edges and/or vertices.  In my view, the
> mechanics are the same but just being performed in a different contexts.
> In the case of MiNiFi, this is effectively an “offline” context.  By
> "offline," I mean we are not strictly bound to loaded components such as
> that in the current NiFi context consisting of items from unpacked NARs.

A unified UI, or at least common code base/components makes a lot of sense.
MiNiFi flows are essentially NiFi flows and so it makes sense to design them
with the same UI.

That said, recent improvements have made it less burdonsome to hand-write the
YAML, and leaving that option makes sense to accomodate users who do not wish
to use a graphical flow designer. I think it's worth exploring further
improvements to the textual configuration landscape.

> How can we minimize the impact to NiFi?

With the emergence of multiple NiFi "execution engines," such as MiNiFi - C++
and MiNiFi - Java, it almost seems to make sense to decouple the UI from NiFi,
leaving the execution engine and UI as separate pieces. The decoupled Flow
Designer would then be the sole, unified tool for graphically designing flows.
This also enables the possibility of using other tools, such as declarative
flow definition files (something like Graphviz dot files, or an improved
version of our YAML syntax) to configure NiFi. This might be a boon to the
product in certain deployment environments/scenarios.

Regards,

Andy I.C.
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.22 (GNU/Linux)

iQEcBAEBAgAGBQJamCr2AAoJEG1+mBKNMpIDqs8H/3OAHIs3QDoQo+dmnhrZTlRY
jCzUBOM2MM9ogs265p3WGNtpSeEH94o6+nCn8eAYpW1g1NZDd93JEvTcUjFPx1Vs
jlgmyOdmqoOZvRb5z4Y2vSjRf76D8QwNE8NMDXQtSxhvshRgOuYpaDxFmoC5dlRI
5XlRKXoHucvpIel2gBS7vEZdeWtc1PBNaxTwbWu+ESjhLEmSD+gtYtTZ/B6oCifI
LkjOCNxRqoA9Du0GDUO7KlOL4ZzcYUI7xNyJFTvzaugIxlX1nAXlgplTz24Pffsc
N2+DLvM0RKmjMINfOEdnArHHA0avQvH8NGj1WCoKUcPWmIKthgTWnDJXtaYLKrk=
=Kagd
-END PGP SIGNATURE-


​Sent from ProtonMail, Swiss-based encrypted email.​

‐‐‐ Original Message ‐‐‐

On February 28, 2018 1:53 AM, Aldrin Piri  wrote:

> ​​
> 
> Hey folks,
> 
> With the release of Registry, I’ve been contemplating leveraging it and the
> 
> current codebase of NiFi to facilitate ease of use for MiNiFi flow design.
> 
> One of the areas I would like to form a concerted effort around is that of
> 
> the Command and Control (C2) functionality originally presented when the
> 
> MiNiFi effort was started and further expounded upon with a feature
> 
> proposal \[1\]. In that proposal, while the names are dated, we have
> 
> components that fulfill some of the core building blocks toward the overall
> 
> vision of C2.
> 
> For those not intimately familiar, MiNiFi is primarily configured via a
> 
> YAML configuration file. \[2\] The initial idea was that flows would be
> 
> simple a source, maybe a transform or two, and then transmission. The
> 
> notion was that hand creation of one of these files would not be overly
> 
> onerous. It became apparent from questions, however, that users were doing
> 
> more complex flows and this YAML ranged from tedious to difficult for those
> 
> situations. This precipitated the best approach currently available,
> 
> utilizing a NiFi instance as a way to generate a template. This template
> 
> was then exported, and using our provided toolkit to get YAML out that is
> 
> deployable with your instance. Better, but far from ideal.
> 
> A couple approaches came to mind to further narrow this gap of user
> 
> expectation and realities of our configuration.
> 
> (1) In our current state, we have heavily relied on convenience of
> 
> component overlap. Generally speaking, we relied on the notion that NiFi’s
> 
> bundled components are a superset of those found in MiNiFi
> 
> implementations. We could have continued that, but this would not
> 
> necessarily free the user of the file handling and transformation
> 
> processes. Additionally, this comes with caveats, among them that given
> 
> the ways in which C++ allows us different accesses, there are some
> 
> interesting processors that work better for that more native approach and
> 
> might not ever justify having an equivalent in Java.
> 
> (2) Providing a first class experience in NiFi for MiNiFi flow designing
> 
> making use of the 

Re: [VOTE] Release Apache NiFi MiNiFi C++ 0.4.0 RC1

[Andy Christianson]

Created https://issues.apache.org/jira/browse/MINIFICPP-384 to track. Should 
have PR shortly.

Sent from [ProtonMail](https://protonmail.com), Swiss-based encrypted email.

 Original Message 
On January 25, 2018 2:46 PM, Andy Christianson <aichr...@protonmail.com> wrote:

> -1 there are flex/bison generated files in extensions/expression-language/
>
> (Scanner.cpp, Parser.hpp, Parser.cpp, *.hh)
>
> This caused build failure on my machine (CentOS 7.4 w/ devtoolset-6) due to 
> incompatibility between generated files and local installation of flex.
>
> We should take these out of the source distribution and allow them to be 
> generated at build time.
>
> Regards,
>
> Andy I.C.
>
> Sent from [ProtonMail](https://protonmail.com), Swiss-based encrypted email.
>
>  Original Message 
> On January 24, 2018 1:36 AM, Aldrin Piri <ald...@apache.org> wrote:
>
>> Hello Apache NiFi Community,
>>
>> I am pleased to call this vote for the source release of Apache NiFi MiNiFi
>> C++, nifi-minifi-cpp-0.4.0.
>>
>> The source archive, signature, and digests can be located at:
>>
>> Source Archive:
>> https://dist.apache.org/repos/dist/dev/nifi/nifi-minifi-cpp/0.4.0/nifi-minifi-cpp-0.4.0-source.tar.gz
>>
>> GPG armored signature:
>> https://dist.apache.org/repos/dist/dev/nifi/nifi-minifi-cpp/0.4.0/nifi-minifi-cpp-0.4.0-source.tar.gz.asc
>>
>> Source MD5:
>> https://dist.apache.org/repos/dist/dev/nifi/nifi-minifi-cpp/0.4.0/nifi-minifi-cpp-0.4.0-source.tar.gz.md5
>>
>> Source SHA1:
>> https://dist.apache.org/repos/dist/dev/nifi/nifi-minifi-cpp/0.4.0/nifi-minifi-cpp-0.4.0-source.tar.gz.sha1
>>
>> Source SHA256:
>> https://dist.apache.org/repos/dist/dev/nifi/nifi-minifi-cpp/0.4.0/nifi-minifi-cpp-0.4.0-source.tar.gz.sha256
>>
>> The Git tag is minifi-cpp-0.4.0-RC1
>> The Git commit hash is c05d467758c861f38a00c6ac5f64f75d6ca0ce05
>>
>> - 
>> https://git-wip-us.apache.org/repos/asf?p=nifi-minifi-cpp.git;a=commit;h=c05d467758c861f38a00c6ac5f64f75d6ca0ce05
>>
>> - 
>> https://github.com/apache/nifi-minifi-cpp/commit/c05d467758c861f38a00c6ac5f64f75d6ca0ce05
>>
>> Checksums of nifi-minifi-cpp-0.4.0-source.tar.gz:
>> MD5: 27d82a635cee3f97841eb6af01248712
>> SHA1: 3b8bb029bcca64897223f7fb27122ac9642cd92b
>> SHA256: 887879d65514ccc7ee598ba650d05f7ff56145d8462707d38b137363ea971965
>>
>> Release artifacts are signed with the following key:
>> https://people.apache.org/keys/committer/aldrin
>>
>> KEYS file available here:
>> https://dist.apache.org/repos/dist/dev/nifi/KEYS
>>
>> 49 issues were closed/resolved for this release:
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12321520=12341641
>>
>> Release note highlights can be found here:
>> https://cwiki.apache.org/confluence/display/MINIFI/Release+Notes#ReleaseNotes-Versioncpp-0.4.0
>>
>> The vote will close 27 January at 12PM EST [1].
>>
>> Please download the release candidate and evaluate the necessary items
>> including checking hashes, signatures, build from source, and test. Then
>> please vote:
>>
>> [ ] +1 Release this package as nifi-minifi-cpp-0.4.0
>> [ ] +0 no opinion
>> [ ] -1 Do not release this package because...
>>
>> Thanks!
>>
>> [1] You can determine this time for your local time zone at
>> https://s.apache.org/minifi-cpp-0.4.0-rc1-close

Re: [VOTE] Release Apache NiFi MiNiFi C++ 0.4.0 RC1

[Andy Christianson]

-1 there are flex/bison generated files in extensions/expression-language/

(Scanner.cpp, Parser.hpp, Parser.cpp, *.hh)

This caused build failure on my machine (CentOS 7.4 w/ devtoolset-6) due to 
incompatibility between generated files and local installation of flex.

We should take these out of the source distribution and allow them to be 
generated at build time.

Regards,

Andy I.C.

Sent from [ProtonMail](https://protonmail.com), Swiss-based encrypted email.

 Original Message 
On January 24, 2018 1:36 AM, Aldrin Piri  wrote:

> Hello Apache NiFi Community,
>
> I am pleased to call this vote for the source release of Apache NiFi MiNiFi
> C++, nifi-minifi-cpp-0.4.0.
>
> The source archive, signature, and digests can be located at:
>
> Source Archive:
> https://dist.apache.org/repos/dist/dev/nifi/nifi-minifi-cpp/0.4.0/nifi-minifi-cpp-0.4.0-source.tar.gz
>
> GPG armored signature:
> https://dist.apache.org/repos/dist/dev/nifi/nifi-minifi-cpp/0.4.0/nifi-minifi-cpp-0.4.0-source.tar.gz.asc
>
> Source MD5:
> https://dist.apache.org/repos/dist/dev/nifi/nifi-minifi-cpp/0.4.0/nifi-minifi-cpp-0.4.0-source.tar.gz.md5
>
> Source SHA1:
> https://dist.apache.org/repos/dist/dev/nifi/nifi-minifi-cpp/0.4.0/nifi-minifi-cpp-0.4.0-source.tar.gz.sha1
>
> Source SHA256:
> https://dist.apache.org/repos/dist/dev/nifi/nifi-minifi-cpp/0.4.0/nifi-minifi-cpp-0.4.0-source.tar.gz.sha256
>
> The Git tag is minifi-cpp-0.4.0-RC1
> The Git commit hash is c05d467758c861f38a00c6ac5f64f75d6ca0ce05
>
> - 
> https://git-wip-us.apache.org/repos/asf?p=nifi-minifi-cpp.git;a=commit;h=c05d467758c861f38a00c6ac5f64f75d6ca0ce05
>
> - 
> https://github.com/apache/nifi-minifi-cpp/commit/c05d467758c861f38a00c6ac5f64f75d6ca0ce05
>
> Checksums of nifi-minifi-cpp-0.4.0-source.tar.gz:
> MD5: 27d82a635cee3f97841eb6af01248712
> SHA1: 3b8bb029bcca64897223f7fb27122ac9642cd92b
> SHA256: 887879d65514ccc7ee598ba650d05f7ff56145d8462707d38b137363ea971965
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/aldrin
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/dev/nifi/KEYS
>
> 49 issues were closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12321520=12341641
>
> Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/MINIFI/Release+Notes#ReleaseNotes-Versioncpp-0.4.0
>
> The vote will close 27 January at 12PM EST [1].
>
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test. Then
> please vote:
>
> [ ] +1 Release this package as nifi-minifi-cpp-0.4.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...
>
> Thanks!
>
> [1] You can determine this time for your local time zone at
> https://s.apache.org/minifi-cpp-0.4.0-rc1-close

Re: MINIFICPP-357 minifi-cpp Design decision: Flow Configuration v3 yml vs properties

[Andy Christianson]

Thanks for the feedback. It looks like we have consensus on keeping instance 
configuration in the properties files and keeping the yml focused on flow.

I'll run some tests to make sure the instance config properties, if present in 
the yml, do not interfere with the instance startup/operations.

Regards,

Andy I.C.

Sent from [ProtonMail](https://protonmail.com), Swiss-based encrypted email.

>  Original Message 
> Subject: Re: MINIFICPP-357 minifi-cpp Design decision: Flow Configuration v3 
> yml vs properties
> Local Time: January 4, 2018 4:22 PM
> UTC Time: January 4, 2018 9:22 PM
> From: kdoran.apa...@gmail.com
> To: dev@nifi.apache.org
>
> Hi Andy,
>
> My preference would also be to keep the .yml scoped to flows, and keep 
> instance config in a .properties file.
>
> When we get further along the C2 roadmap, both of these should be updatable 
> via C2 interactions.
>
> Thanks,
> Kevin
>
> On 1/4/18, 16:04, "Joe Witt" joe.w...@gmail.com wrote:
>
> Hello
>
> My preference is that we follow similar logic to how NiFi has
> 'instance configuration' details in nifi.properties and 'flow
> configuration' details in flow.xml.gz.
>
> What we do in minificpp will hopefully be aligned with what we do in
> minifi-java and hopefully minifi-java just becomes an assembly of
> nifi.
>
> Thanks
> Joe
>
> On Thu, Jan 4, 2018 at 3:59 PM, Aldrin Piri aldrinp...@gmail.com wrote:
>
>> For the time being, I kind of view the C++ variant's configuration to be a
>
>> subset of that which is available for Java.  Many of these items currently
>
>> have no backing, so don't know that we need to provide explicit support for
>
>> them at this juncture but ensure that their presence does not preclude
>
>> proper operation of a C++ instance.
>
>>
>
>> Looking at things from a general handling standpoint, I see these as
>
>> considerations of flow vs instance configuration.  My general inclination
>
>> is to provide such separation at some point and is at the heart of
>
>> MINIFI-66 [1].  I think what is important regardless of how this carries
>
>> out, and in view of the first release of Registry, is to think how we can
>
>> make use of what facilities Registry provides both now in terms of flows as
>
>> well as future resource types.  Some combination of the command and control
>
>> work (both server and the initial C2 API in C++) with Registry should allow
>
>> us to manage both sets of configuration that map to and enable the most
>
>> common workflows users would expect.  I suspect the C++ approach is more
>
>> aligned with typical usage, properties generally bootstrapped in
>
>> .properties file(s), with some C2 API type interactions that may allow
>
>> adjusting these items in a separate context outside of the flow itself.
>
>>
>
>> [1] https://issues.apache.org/jira/browse/MINIFI-66
>
>>
>
>> On Thu, Jan 4, 2018 at 1:27 PM, Andy Christianson <aichr...@protonmail.com>
>
>> wrote:
>
>>
>
>>> All,
>
>>>
>
>>> Currently in the middle of MINIFICPP-357.
>
>>>
>
>>> v3 includes core/repository configuration in the yml file, while
>
>>> traditionally in minificpp this has been set with the .properties files in
>
>>> config/.
>
>>>
>
>>> E.g.
>
>>>
>
>>> Core Properties:
>
>>>
>
>>> flow controller graceful shutdown period: 10 sec
>
>>>
>
>>> flow service write delay interval: 500 ms
>
>>>
>
>>> administrative yield duration: 30 sec
>
>>>
>
>>> bored yield duration: 10 millis
>
>>>
>
>>> max concurrent threads: 1
>
>>>
>
>>> variable registry properties: ''
>
>>>
>
>>> FlowFile Repository:
>
>>>
>
>>> partitions: 256
>
>>>
>
>>> checkpoint interval: 2 mins
>
>>>
>
>>> always sync: false
>
>>>
>
>>> Swap:
>
>>>
>
>>> threshold: 2
>
>>>
>
>>> in period: 5 sec
>
>>>
>
>>> in threads: 1
>
>>>
>
>>> out period: 5 sec
>
>>>
>
>>> out threads: 4
>
>>>
>
>>> Content Repository:
>
>>>
>
>>> content claim max appendable size: 10 MB
>
>>>
>
>>> content claim max flow files: 100
>
>>>
>
>>> always sync: false
>
>>>
>
>>> How should we handle this in minificpp with v3 support? Do we want to keep
>
>>> the convention of using the .properties file, or move all that into the yml?
>
>>>
>
>>> Regards,
>
>>>
>
>>> Andy I.C.
>
>>>
>
>>> Sent from [ProtonMail](https://protonmail.com), Swiss-based encrypted
>
>>> email.

MINIFICPP-357 minifi-cpp Design decision: Flow Configuration v3 yml vs properties

[Andy Christianson]

All,

Currently in the middle of MINIFICPP-357.

v3 includes core/repository configuration in the yml file, while traditionally 
in minificpp this has been set with the .properties files in config/.

E.g.

Core Properties:

flow controller graceful shutdown period: 10 sec

flow service write delay interval: 500 ms

administrative yield duration: 30 sec

bored yield duration: 10 millis

max concurrent threads: 1

variable registry properties: ''

FlowFile Repository:

partitions: 256

checkpoint interval: 2 mins

always sync: false

Swap:

threshold: 2

in period: 5 sec

in threads: 1

out period: 5 sec

out threads: 4

Content Repository:

content claim max appendable size: 10 MB

content claim max flow files: 100

always sync: false

How should we handle this in minificpp with v3 support? Do we want to keep the 
convention of using the .properties file, or move all that into the yml?

Regards,

Andy I.C.

Sent from [ProtonMail](https://protonmail.com), Swiss-based encrypted email.

Re: [VOTE] Release Apache NiFi MiNiFI C++ 0.3.0 (RC2)

[Andy Christianson]

+1

Sent from [ProtonMail](https://protonmail.com), Swiss-based encrypted email.

>  Original Message 
> Subject: [VOTE] Release Apache NiFi MiNiFI C++ 0.3.0 (RC2)
> Local Time: November 27, 2017 11:59 AM
> UTC Time: November 27, 2017 4:59 PM
> From: phroc...@apache.org
> To: dev@nifi.apache.org
>
> Hello Apache NiFi Community,
>
> I am pleased to call this vote for the source release of Apache NiFi MiNiFi
> C++, nifi-minifi-cpp-0.3.0.
>
> The source archive, signature, and digests can be located at:
>
> Source Archive:
>
> https://dist.apache.org/repos/dist/dev/nifi/nifi-minifi-cpp/0.3.0/nifi-minifi-cpp-0.3.0-source.tar.gz
> GPG armored signature:
>
> https://dist.apache.org/repos/dist/dev/nifi/nifi-minifi-cpp/0.3.0/nifi-minifi-cpp-0.3.0-source.tar.gz.asc
> Source MD5:
>
> https://dist.apache.org/repos/dist/dev/nifi/nifi-minifi-cpp/0.3.0/nifi-minifi-cpp-0.3.0-source.tar.gz.md5
> Source SHA1:
>
> https://dist.apache.org/repos/dist/dev/nifi/nifi-minifi-cpp/0.3.0/nifi-minifi-cpp-0.3.0-source.tar.gz.sha1
> Source SHA256:
>
> https://dist.apache.org/repos/dist/dev/nifi/nifi-minifi-cpp/0.3.0/nifi-minifi-cpp-0.3.0-source.tar.gz.sha256
>
> The Git tag is minifi-cpp-0.3.0-RC2
> The Git commit hash is 0e4ce79b78cbc9c89189eabb8042749a873d9723
> *
> https://git-wip-us.apache.org/repos/asf?p=nifi-minifi-cpp.git;a=commit;h=0e4ce79b78cbc9c89189eabb8042749a873d9723
> *
> https://github.com/apache/nifi-minifi-cpp/commit/0e4ce79b78cbc9c89189eabb8042749a873d9723
>
> Checksums of nifi-minifi-cpp-0.3.0-source.tar.gz:
> MD5: aeb800c4bc8829fe8af300e75d2cd15b
> SHA1: 6dbc094ba876c7120ceb6a96d19ae8d37c62d662
> SHA256: 1fffb6697c3cff6dd4b9ceb7c91a575df149acd160315aeb9c8a55f20d159372
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/phrocker
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/dev/nifi/KEYS
>
> 61 issues were closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12321520=12341640
>
> Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/MINIFI/Release+Notes#ReleaseNotes-Versioncpp-0.3.0
>
> The vote will be open for 72 hours and will close 30 Nov at 12PM EDT [1].
>
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test. Then
> please vote:
>
> [ ] +1 Release this package as nifi-minifi-cpp-0.3.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...
>
> Thanks!
>
> [1] You can determine this time for your local time zone at
> https://s.apache.org/minifi-cpp-0.3.0-rc2-close

Re: [ANNOUNCE] New NiFi PMC member Mike Moser

[Andy Christianson]

Congrats!

Sent from [ProtonMail](https://protonmail.com), Swiss-based encrypted email.

>  Original Message 
> Subject: [ANNOUNCE] New NiFi PMC member Mike Moser
> Local Time: November 20, 2017 4:18 PM
> UTC Time: November 20, 2017 9:18 PM
> From: tk...@apache.org
> To: dev@nifi.apache.org
>
> NiFi community,
> On behalf of the Apache NiFi PMC, I am pleased to announce that Mike Moser
> has accepted the PMC's invitation to join the Apache NiFi PMC. We greatly
> appreciate all of Mike's hard work and generous contributions to the
> project. We look forward to continued involvement in the project.
>
> Mike has been a long time code contributor to NiFi, he performed the final
> release management duties for the 0.x line, has been involved with
> verifying releases and always a delight on the mailing lists.
>
> Please join us in congratulating and welcoming Mike to the Apache NiFi PMC.
>
> Congratulations and welcome, Mike!
>
> -- Tony

Re: [ANNOUNCE] New Apache NiFi Committer Mike Hogue

[Andy Christianson]

Grats!

Sent from ProtonMail, Swiss-based encrypted email.


> Original Message 
>Subject: Re: [ANNOUNCE] New Apache NiFi Committer Mike Hogue
>Local Time: November 10, 2017 6:05 PM
>UTC Time: November 10, 2017 11:05 PM
>From: phroc...@apache.org
>To: dev@nifi.apache.org
>
>Congrats Mike!
>
> On Fri, Nov 10, 2017 at 8:33 AM, Vincent Russell vincent.russ...@gmail.com
> wrote:
>>congrats Mike
>>On Fri, Nov 10, 2017 at 8:20 AM, Matt Burgess mattyb...@apache.org
>> wrote:
>>>Congratulations Mike, glad to have you aboard!
>>>On Thu, Nov 9, 2017 at 7:14 PM, Tony Kurc tk...@apache.org wrote:
On behalf of the Apache NiFI PMC, I am very pleased to announce that
 Mike
>Hogue has accepted the PMC's invitation to become a committer on the
 Apache
 NiFi project. We greatly appreciate all of Mike's hard work and
 generous
>contributions to the project. We look forward to his continued
 involvement
 in the project.
In the past several months, Mike has contributed on a number of fronts.
 He
 was primary author on several new features in 1.4.0 to include the gRPC
 bundle. He also improved SSL/TLS controller services, and made great
 documentation improvements on LICENSE and NOTICE best practices and has
 helped work down our pull request backlog by reviewing new
 contributions.
>Welcome and congratulations!
Tony

Re: [DISCUSS] Changing core components of MiNiFi to support compartmentalization

[Andy Christianson]

Marc,

Thanks for putting this design doc together. Here are some thoughts:

- Multiple INCLUDE_ONLY statements is internally contradictory. If we go this 
route, we should allow INCLUDE_ONLY to take a list.

More broadly, I think we need to uniformly support either a white list or a 
black list of features. In master right now, it is more of a blacklist (you 
DISABLE what you don't want, everything ENABLEd by default). Given the nature 
and purpose of minifi, switching to a whitelist (ENABLE everything you want 
included; everything else is DISABLEd by default).

Instead of INCLUDE_ONLY, I think we should have a simple, consistent ENABLE_* 
cmake interface. To make this a little less brutal on end-users, we may want to 
borrow existing package management ideas, such as groups (denoted here with an 
ENABLE_ALL_ pattern):

- ENABLE_ALL_CORE_PROCESSORS
- ENABLE_ALL_ARCHIVE_PROCESSORS
- ENABLE_ALL_USB_PROCESSORS
- ENABLE_ALL

Enabling a group would enable everything under it. For the user who wishes to 
include everything, a simple catch-all ENABLE_ALL would be provided.

Regarding the public libminifi interface, I think we should provide two: 
libminifi.so/libminifi.a libminifi++.so/libminifi++.a. The C-interface would be 
the lowest-common-denominator, and the ++ (c++) interface would just publicly 
expose the relevant classes as they exist.

Given libminifi is currently C++, if we stick with that internally, the C 
interface should just wrap and translate the C++ interfaces/objects and 
hopefully not add additional constraints or complexity to the internal design. 
If our aim is to support environments where only a C compiler is available, 
then there needs to be a very clear delineation in the code base of where C 
begins and C++ ends.

Regards,

Andy I.C.

Sent from ProtonMail, Swiss-based encrypted email.


> Original Message 
>Subject: [DISCUSS] Changing core components of MiNiFi to support 
>compartmentalization
>Local Time: November 9, 2017 10:54 AM
>UTC Time: November 9, 2017 3:54 PM
>From: phroc...@apache.org
>To: dev@nifi.apache.org
>
>Hi,
> I've put together a proposal [1] and would like some feedback before I
> begin these changes. The goal is to support a more fluid API experience
> with the addition of C function calls to interact with libminifi. The goal
> of this article is to discuss some initial steps to getting a usable API
> from the build target we have known as libminifi.
>
> I would love to hear any feedback. If we take this route we open the door
> for many people who want the externalized C lib; however, a C interface to
> C++ may not be for everyone.
>
> Should we simply provide a C library or is this good enough for everyone?
> These are questions that I ask to the community so we can make an informed
> decision.
>
> I've laid out a few pros and cons. Performance degradation, real or
> imagined, will surely be the biggest con anyone can lay out, but would
> appreciate more input in the comment section of the proposal.
>
> Thanks,
> Marc
>
> [1] https://cwiki.apache.org/confluence/display/MINIFI/
> LibMiNiFi+Design+Proposal
>

minifi-cpp: Expression language core API

[Andy Christianson]

MiNiFi - C++ Devs,

I am currently working on MINIFICPP-49, the expression language feature. While 
expression compilation and evaluation is fairly self-contained, at the very 
least the API to access expression evaluation will touch core components.

Here is how NiFi is currently exposing expression evaluation to processors:

...
try {
// read the url property from the context
final String urlstr = 
trimToEmpty(context.getProperty(PROP_URL).evaluateAttributeExpressions(requestFlowFile).getValue());
final URL url = new URL(urlstr);
...

While we have the opportunity now to improve this, we have a couple design 
constraints: the expression code comes from properties, and dynamic evaluation 
of it requires a flow file as input.

Because expressions are defined as processor properties, it is natural to 
expose expression evaluation via the ProcessContext API. The current minifi-cpp 
API to get static properties is as follows:

bool getProperty(const std::string , std::string ) {
  return processor_node_->getProperty(name, value);
}

If we do not wish to introduce a Property type with its own 
evaluateAttributeExpressions method, we could simply introduce another 
ProcessContext method for evaluating dynamic properties:

bool evaluateProperty(const std::string , const core::FlowFile 
_file, std::string ) {
  ...
}

The implementation of this would compile the expression (the raw value as 
returned by getProperty(...)) if it has not yet been compiled, then evaluate 
the compiled expression against the provided FlowFile. The end result is an API 
similar to, but simpler than, the NiFi interface. The alternative is to provide 
the expression primitives to processors and allow them to manage 
compilation/evaluation on their own. This would increase complexity across all 
processors which support expression properties, which will likely be most 
processors.

The next important question which impacts core minifi is whether or not 
expression language should be an extension. Whether or not it is an extension, 
some kind of standard interface to expressions will need to be made available 
to all processors. Here are the pros/cons of putting it in an extension, as far 
as I can tell:

Pros:

- Reduce compiled size of minifi somewhat (the lexer/parser is currently ~4300 
lines of C++ with no additional library or runtime dependencies) when feature 
is disabled
- Allow for alternate expression language implementations in the future

Cons:

- Additional complexity by needing to add Expression primitives, a standard 
Expression compiler API, dynamic object loading, and an empty (NoOp) 
implementation if the extension is not included
- Additional vtable lookups on an operation which will be invoked very 
frequently (every property lookup on every flow file which supports expressions)
- Makes it harder for gcc/clang/etc. to inline/optimize expression language 
functions
- Core processors (e.g. GetFile/PutFile, where expression language will almost 
certainly be desired for file paths and other properties) will depend on an 
optional extension

I would like to hear feedback from the dev community on these two important 
topics (the interface to the expression language and whether or not the 
implementation should be an extension) before writing the code that touches 
core components. The API question is ultimately more important because it 
touches all current and future processor authors. The decision of whether it is 
an extension or not is more reversible.

Regards,

Andy I.C.

Re: Issue with Executescript

[Andy Christianson]

Hi Vyshali,

Follow the usual Java String API [1]. Instead of splitlines, use .split(String 
regex).

-Andy

[1]: https://docs.oracle.com/javase/6/docs/api/java/lang/String.html

From: Vyshali 
Sent: Thursday, November 2, 2017 2:14 PM
To: dev@nifi.apache.org
Subject: Re: Issue with Executescript

Thank you Andy.

How can I convert the "text" into a list or array? So that I could get rid
of splitlines funtion itself

Regards,
Vyshali



--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

Re: Issue with Executescript

[Andy Christianson]

Vyshali,

Please also consider looking at your splitlines call. Splitlines is defined in 
python's script class, but not in Java. This is Jython, so IOUtils.toString 
returns a Java string, not a Python string, and that string splits in a 
different way.

Regards,

Andy

From: Andy Christianson <achristian...@hortonworks.com>
Sent: Thursday, November 2, 2017 1:40 PM
To: dev@nifi.apache.org
Subject: Re: Issue with Executescript

Vyshali,

These types of issues can be difficult to diagnose in a running NiFi instance. 
Once your script reaches beyond a certain threshold, I recommend creating unit 
tests for your script and mocking the NiFi interfaces. MattyB's script tester 
[1] may help you with this task as well as general script testing.

My initial eyeball of your code tells me there's something up with your 
putattribute.

On a final note, if you're not familiar with Python, are there any other 
languages you are more familiar with? ExecuteScript allows you to use almost 
any language you could think of, so if you are more comfortable with another 
supported language, I recommend porting to that. Between using a language 
your'e more comfortable with, and using MattyB's script tester, I'm confident 
we should be able to resolve the issue.

Regards,

Andy

P.S. I would consider this more of a user question than a developer question, 
so next time please email us...@nifi.apache.org rather than the dev list.

[1]: 
http://funnifi.blogspot.com/2017/10/release-120-of-nifi-script-tester.html?m=1

From: Vyshali <vyshal...@honeywell.com>
Sent: Thursday, November 2, 2017 1:31 PM
To: dev@nifi.apache.org
Subject: Issue with Executescript

Hi,

I'm using the executescript process to generate some fake data using "Faker"
package and replacing it in the original data.I have attached the script for
your reference.

import java.io
from org.apache.commons.io import IOUtils
from java.nio.charset import StandardCharsets
from org.apache.nifi.processor.io import StreamCallback
import unicodecsv as csv
from faker import Factory
from collections import defaultdict

class TransformCallback(StreamCallback):
def _init_(self):
pass

def process(self,inputStream,outputStream):
text = IOUtils.toString(inputStream,StandardCharsets.ISO_8859_1)
faker  = Factory.create()//generating fake data
names  = defaultdict(faker.name)
emails = defaultdict(faker.email)
ssns = defaultdict(faker.ssn)
phone_numbers = defaultdict(faker.phone_number)

for row in text.splitlines():
row["name"]  = names[row["name"]] //Assigning the fake data
row["email"] = emails[row["email"]]
row["ssn"] = ssns[row["ssn"]]
row["phone_number"] = phone_numbers[row["phone_number"]]
flowFile = session.putAttribute(flowFile,"name",row["name"])

outputStream.write(text.encode('UTF8'))


flowFile = session.get()
if flowFile != None:
flowFile = session.write(flowFile,TransformCallback())
session.transfer(flowFile, REL_SUCCESS)
session.commit()

But I'm unable to execute it successfully.I'm getting the following error
"ProcessException:TypeError:None required"

I'm not much familiar to python.Please give me suggestions on how can I
solve this.Correct me in case my coding is also not appropriate.

Regards,
Vyshali




--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

Re: Issue with Executescript

[Andy Christianson]

Vyshali,

These types of issues can be difficult to diagnose in a running NiFi instance. 
Once your script reaches beyond a certain threshold, I recommend creating unit 
tests for your script and mocking the NiFi interfaces. MattyB's script tester 
[1] may help you with this task as well as general script testing.

My initial eyeball of your code tells me there's something up with your 
putattribute.

On a final note, if you're not familiar with Python, are there any other 
languages you are more familiar with? ExecuteScript allows you to use almost 
any language you could think of, so if you are more comfortable with another 
supported language, I recommend porting to that. Between using a language 
your'e more comfortable with, and using MattyB's script tester, I'm confident 
we should be able to resolve the issue.

Regards,

Andy

P.S. I would consider this more of a user question than a developer question, 
so next time please email us...@nifi.apache.org rather than the dev list.

[1]: 
http://funnifi.blogspot.com/2017/10/release-120-of-nifi-script-tester.html?m=1

From: Vyshali 
Sent: Thursday, November 2, 2017 1:31 PM
To: dev@nifi.apache.org
Subject: Issue with Executescript

Hi,

I'm using the executescript process to generate some fake data using "Faker"
package and replacing it in the original data.I have attached the script for
your reference.

import java.io
from org.apache.commons.io import IOUtils
from java.nio.charset import StandardCharsets
from org.apache.nifi.processor.io import StreamCallback
import unicodecsv as csv
from faker import Factory
from collections import defaultdict

class TransformCallback(StreamCallback):
def _init_(self):
pass

def process(self,inputStream,outputStream):
text = IOUtils.toString(inputStream,StandardCharsets.ISO_8859_1)
faker  = Factory.create()//generating fake data
names  = defaultdict(faker.name)
emails = defaultdict(faker.email)
ssns = defaultdict(faker.ssn)
phone_numbers = defaultdict(faker.phone_number)

for row in text.splitlines():
row["name"]  = names[row["name"]] //Assigning the fake data
row["email"] = emails[row["email"]]
row["ssn"] = ssns[row["ssn"]]
row["phone_number"] = phone_numbers[row["phone_number"]]
flowFile = session.putAttribute(flowFile,"name",row["name"])

outputStream.write(text.encode('UTF8'))


flowFile = session.get()
if flowFile != None:
flowFile = session.write(flowFile,TransformCallback())
session.transfer(flowFile, REL_SUCCESS)
session.commit()

But I'm unable to execute it successfully.I'm getting the following error
"ProcessException:TypeError:None required"

I'm not much familiar to python.Please give me suggestions on how can I
solve this.Correct me in case my coding is also not appropriate.

Regards,
Vyshali




--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

Re: minifi-cpp - Consensus regarding enablement of fsanitize=address

[Andy Christianson]

PR is in [1].

Regards,

Andy I.C.

[1]: https://github.com/apache/nifi-minifi-cpp/pull/154

From: Andy Christianson <achristian...@hortonworks.com>
Sent: Wednesday, October 25, 2017 10:47 AM
To: dev@nifi.apache.org
Subject: Re: minifi-cpp - Consensus regarding enablement of fsanitize=address

Sounds good. I have a local commit which I'll move to a branch and submit a PR 
for, after adding a relevant JIRA.

Regards,

Andy I.C.

From: Marc <phroc...@apache.org>
Sent: Wednesday, October 25, 2017 8:08 AM
To: dev@nifi.apache.org
Subject: Re: minifi-cpp - Consensus regarding enablement of fsanitize=address

Andy,
  I agree with your suggestion. There's no reason why we couldn't make this
an option in the future after its removal to help clean up DEBUG.

  Thanks,
  Marc

On Tue, Oct 24, 2017 at 9:48 AM, Andy Christianson <
achristian...@hortonworks.com> wrote:

> MiNiFi - C++ Devs:
>
>
> On an earlier commit, I added the following to the root CMakeLists.txt:
>
>
> # Enable asan in DEBUG for compatibility with civet
> set(CMAKE_CXX_FLAGS_DEBUG "${CMAKE_CXX_FLAGS_DEBUG} -fsanitize=address")
>
>
> This was, as stated in the comment, to address a compatibility issue when
> compiling in DEBUG where civet would fail to link.
>
>
> On more recent dev toolchains (devtoolset-6 SCL CentOS/GCC 6.3), this
> results in extremely verbose and unhelpful output during test runs. While I
> am not opposed to using these tools to detect legitimate memory leaks, as
> it stands currently the setting is more of a detriment to the development
> process. I think that we should either only turn this on when we
> deliberately intend to analyze memory usage, or possibly create additional
> unit tests to detect memory leaks.
>
>
> In the meantime, I think we should remove the statement from our
> CMakeLists.txt and disable it in the CMakeLists.txt for civet as well.
>
>
> I would like to hear additional feedback from the development community to
> see if we have consensus on this change.
>
>
> Regards,
>
>
> Andy I.C.
>

Re: minifi-cpp - Consensus regarding enablement of fsanitize=address

[Andy Christianson]

Sounds good. I have a local commit which I'll move to a branch and submit a PR 
for, after adding a relevant JIRA.

Regards,

Andy I.C.

From: Marc <phroc...@apache.org>
Sent: Wednesday, October 25, 2017 8:08 AM
To: dev@nifi.apache.org
Subject: Re: minifi-cpp - Consensus regarding enablement of fsanitize=address

Andy,
  I agree with your suggestion. There's no reason why we couldn't make this
an option in the future after its removal to help clean up DEBUG.

  Thanks,
  Marc

On Tue, Oct 24, 2017 at 9:48 AM, Andy Christianson <
achristian...@hortonworks.com> wrote:

> MiNiFi - C++ Devs:
>
>
> On an earlier commit, I added the following to the root CMakeLists.txt:
>
>
> # Enable asan in DEBUG for compatibility with civet
> set(CMAKE_CXX_FLAGS_DEBUG "${CMAKE_CXX_FLAGS_DEBUG} -fsanitize=address")
>
>
> This was, as stated in the comment, to address a compatibility issue when
> compiling in DEBUG where civet would fail to link.
>
>
> On more recent dev toolchains (devtoolset-6 SCL CentOS/GCC 6.3), this
> results in extremely verbose and unhelpful output during test runs. While I
> am not opposed to using these tools to detect legitimate memory leaks, as
> it stands currently the setting is more of a detriment to the development
> process. I think that we should either only turn this on when we
> deliberately intend to analyze memory usage, or possibly create additional
> unit tests to detect memory leaks.
>
>
> In the meantime, I think we should remove the statement from our
> CMakeLists.txt and disable it in the CMakeLists.txt for civet as well.
>
>
> I would like to hear additional feedback from the development community to
> see if we have consensus on this change.
>
>
> Regards,
>
>
> Andy I.C.
>

Re: Change of persistent repo in MiNiFi C++

[Andy Christianson]

Mark,

Nice work on the port to RocksDB. Are there any positive/negative runtime 
performance or resource usage implications of the change?

Regards,

Andy

On 10/17/17, 1:28 PM, "Marc"  wrote:

Hello,
 In preparation of solidifying some of our persistent storage code we've
moved from LevelDB to RocksDB as the default persistent storage mechanism.
There should be little change for you other than the installation of
RocksDB. The net benefit will be improved reliability and maintainability
of the dependency's code base.

If you do not wish to install this dependency or your distro doesn't have
RocksDB available in a public repo, our build process will use the version
provided in the third party directory and include the library into the
static library. This will not install RocksDB, but will increase build
times when RocksDB needs to be rebuilt. It is advised that you install
RocksDB from source or your distro's public repos.

To install RocksDB

Ubuntu

sudo apt-get install librocksdb4.1 librocksdb-dev

MacOSX

brew install rocksdb

For CENTOS there does not appear to be a public repo with RocksDB, so
please follow the instructions the RocksDB install to install it
permanently and avoid long build times.

https://github.com/facebook/rocksdb/blob/master/INSTALL.md

Finally, if you wish to avoid using system libraries and build our version
of RocksDB, you can always specify BUILD_ROCKSDB=ON as a variable to CMAKE.
You can also DISABLE_ROCKSDB , but this will default to using volatile
repositories.
Please let me know if you have any questions.
Marc

MINIFI-398 PR

[Andy Christianson]

MiNiFi - C++ devs,

I have submitted a PR for MINIFI-398 [1]. The notable addition is a
test_s2s which leverages the docker system integration test framework to
deploy a minificpp instance, a nifi instance, then send data through
them using the s2s protocol.

Please review and merge at your earliest convenience.

-Andy I.C.


[1] https://github.com/apache/nifi-minifi-cpp/pull/136

Re: Separate MiNiFi projects in JIRA

[Andy Christianson]

Glad to hear it! Thanks for taking the time.

-Andy I.C.

On 9/19/17, 9:57 AM, "Aldrin Piri" <aldrinp...@gmail.com> wrote:

Hey folks,

The new JIRA project is now live.  Please make use of that when filing C++
related JIRAs.  I am going to start moving appropriate items to that
instance.

On Mon, Sep 4, 2017 at 2:13 PM, Aldrin Piri <aldrinp...@gmail.com> wrote:

> Hey folks,
>
> I entered an issue (https://issues.apache.org/jira/browse/MINIFI-397) to
> get this done and will initiate the associated ticket(s) with INFRA to 
make
> this happen.
>
> --aldrin
    >
> On Tue, Aug 22, 2017 at 12:00 PM, Andy Christianson <
> achristian...@hortonworks.com> wrote:
>
>> +1
>>
>> On 8/22/17, 11:57 AM, "Kevin Doran" <kdoran.apa...@gmail.com> wrote:
>>
>> Clones can cross projects. I'm a +1 for the suggestion of separate
>> projects so as to keep a 1-to-1 between projects and repos. Related 
tickets
>> can be linked or cloned to provide context when applicable.
>>
>> Thanks,
>> Kevin
>>
>> On 8/22/17, 11:45, "Jeff Zemerick" <jzemer...@apache.org> wrote:
>>
>> When I briefly looked through the tickets last week none stood
>> out to me as
>> applying to both projects. Granted, some potentially could like
>> changing
>> the Docker base image. With pull requests and GitHub I am of the
>> opinion
>> there should be a one-to-one-to-one correlation between ticket,
>> pull
>> request, and project. I know you can Clone a ticket but I don't
>> know if
>> it's possible to move the clone to a different project.
>>
>> On Tue, Aug 22, 2017 at 11:38 AM, Tony Kurc <trk...@gmail.com>
>> wrote:
>>
>> > If there is a ticket that applies to multiple implementations,
>> separate
>> > jira projects makes that a bit more complicated. How often is
>> that likely
>> > to happen?
>> >
>> > On Tue, Aug 22, 2017 at 10:44 AM, Joe Witt <joe.w...@gmail.com>
>> wrote:
>> >
>> > > Since changing the permissions on requirement for a given
>> field and
>> > > creating a new JIRA project both require ASF infra (i
>> believe) then
>> > > perhaps we should just go with the JIRA project route as that
>> is
>> > > cleaner/easier in the long run.
>> > >
>> > > What do ya'll think?
>> > >
>> > > On Tue, Aug 22, 2017 at 10:33 AM, Kevin Doran <
>> kdoran.apa...@gmail.com>
>> > > wrote:
>> > > > I agree that would be an improvement to my suggestion of
>> making the
>> > > existing Component field required. As to feasibility, I leave
>> that up to
>> > > someone that has more experience working with ASF infra to
>> administer
>> > these
>> > > ASF JIRA projects (Aldrin?).
>> > > >
>> > > > -Kevin
>> > > >
>> > > > On 8/21/17, 15:00, "Jeff Zemerick" <jzemer...@apache.org>
>> wrote:
>> > > >
>> > > > Would it be possible to use a JIRA custom field (that's
>> required)
>> > > called
>> > > > "Implementation" or something similarly named with
>> choices of C++
>> > > and Java?
>> > > > With more than just Java and C++ for components I'm
>> afraid those
>> > two
>> > > > choices might be overlooked when a ticket is created.
>> > > >
>> > > > On Mon, Aug 21, 2017 at 11:37 AM, Andy Christianson <
>> > > > achristian...@hortonworks.com> wrote:
>> > > >
>> > > > > Making it required sounds like an improvement, at the
>> very least.
>> > > > >
>> > > > > -Andy I.C.
>>

MINIFI-389 PR

[Andy Christianson]

MiNiFi cpp devs,


I have submitted a PR [1] which closes out MINIFI-389. Please review & merge at 
your earliest convenience. The tests for MINIFI-389 will be included as part of 
MINIFI-374.


[1] https://github.com/apache/nifi-minifi-cpp/pull/132?


Regards,


Andy I.C.

Re: Separate MiNiFi projects in JIRA

[Andy Christianson]

+1

On 8/22/17, 11:57 AM, "Kevin Doran" <kdoran.apa...@gmail.com> wrote:

Clones can cross projects. I'm a +1 for the suggestion of separate projects 
so as to keep a 1-to-1 between projects and repos. Related tickets can be 
linked or cloned to provide context when applicable.

Thanks,
Kevin

On 8/22/17, 11:45, "Jeff Zemerick" <jzemer...@apache.org> wrote:

When I briefly looked through the tickets last week none stood out to 
me as
applying to both projects. Granted, some potentially could like changing
the Docker base image. With pull requests and GitHub I am of the opinion
there should be a one-to-one-to-one correlation between ticket, pull
request, and project. I know you can Clone a ticket but I don't know if
it's possible to move the clone to a different project.

On Tue, Aug 22, 2017 at 11:38 AM, Tony Kurc <trk...@gmail.com> wrote:

> If there is a ticket that applies to multiple implementations, 
separate
> jira projects makes that a bit more complicated. How often is that 
likely
> to happen?
>
> On Tue, Aug 22, 2017 at 10:44 AM, Joe Witt <joe.w...@gmail.com> wrote:
>
> > Since changing the permissions on requirement for a given field and
> > creating a new JIRA project both require ASF infra (i believe) then
> > perhaps we should just go with the JIRA project route as that is
> > cleaner/easier in the long run.
> >
> > What do ya'll think?
> >
> > On Tue, Aug 22, 2017 at 10:33 AM, Kevin Doran 
<kdoran.apa...@gmail.com>
> > wrote:
> > > I agree that would be an improvement to my suggestion of making 
the
> > existing Component field required. As to feasibility, I leave that 
up to
> > someone that has more experience working with ASF infra to 
administer
> these
> > ASF JIRA projects (Aldrin?).
> > >
> > > -Kevin
> > >
> > > On 8/21/17, 15:00, "Jeff Zemerick" <jzemer...@apache.org> wrote:
> > >
> > > Would it be possible to use a JIRA custom field (that's 
required)
> > called
> > > "Implementation" or something similarly named with choices of 
C++
> > and Java?
> > > With more than just Java and C++ for components I'm afraid 
those
> two
> > > choices might be overlooked when a ticket is created.
> > >
> > > On Mon, Aug 21, 2017 at 11:37 AM, Andy Christianson <
> > > achristian...@hortonworks.com> wrote:
> > >
> > > > Making it required sounds like an improvement, at the very 
least.
> > > >
> > > > -Andy I.C.
> > > > 
> > > > From: Kevin Doran <kdoran.apa...@gmail.com>
> > > > Sent: Monday, August 21, 2017 11:22 AM
> > > > To: dev@nifi.apache.org
> > > > Subject: Re: Separate MiNiFi projects in JIRA
> > > >
> > > > Would  it suffice to make the existing 'component'  field
> > _required_ at
> > > > ticket creation time, and having components consist of 
'C++',
> > 'Java', &
    > > > > perhaps 'Both/All/*' as well? I imagine that is less effort 
than
> > setting up
> > > > and maintaining a separate project and solves the problem, 
unless
> > there are
> > > > advantages that a separate project would provide other than 
just
> > issue
> > > > filtering by C++/Java.
> > > >
> > > > Kevin
> > > >
> > > > On 8/21/17, 11:18, "Andy Christianson" <
> > achristian...@hortonworks.com>
> > > > wrote:
> > > >
> > > > Joe,
> > > >
> > > > We actually already have that. There is a 'C++' and 
'Java'
> > component.
> > > > It works for the most part, but there are cases where it 
becomes
> > ambiguous,
> > > > particularl

Re: Separate MiNiFi projects in JIRA

[Andy Christianson]

Making it required sounds like an improvement, at the very least.

-Andy I.C.

From: Kevin Doran <kdoran.apa...@gmail.com>
Sent: Monday, August 21, 2017 11:22 AM
To: dev@nifi.apache.org
Subject: Re: Separate MiNiFi projects in JIRA

Would  it suffice to make the existing 'component'  field _required_ at ticket 
creation time, and having components consist of 'C++', 'Java', & perhaps 
'Both/All/*' as well? I imagine that is less effort than setting up and 
maintaining a separate project and solves the problem, unless there are 
advantages that a separate project would provide other than just issue 
filtering by C++/Java.

Kevin

On 8/21/17, 11:18, "Andy Christianson" <achristian...@hortonworks.com> wrote:

Joe,

We actually already have that. There is a 'C++' and 'Java' component. It 
works for the most part, but there are cases where it becomes ambiguous, 
particularly on docker-related tickets.

I think there's certainly an argument that we need to just track components 
more carefully. Having it be a separate JIRA would make it harder to make a 
ticket ambiguous. Is it worth the effort/overhead of setting up another JIRA? 
I'll leave that to the more experienced/established Apache parties since I 
don't know what the overhead cost is.

Regards,

Andy I.C.

From: Joe Witt <joe.w...@gmail.com>
Sent: Monday, August 21, 2017 11:10 AM
To: dev@nifi.apache.org
Subject: Re: Separate MiNiFi projects in JIRA

Can we recommend and setup a set of component names so that filtering
can be done reasonably?

If we do that would it be sufficient?

Alternatively we can ask ASF infra to setup another JIRA project such
as 'minificpp' but I'd like to avoid that until we're really sure we
want to bug em.

Thanks

On Mon, Aug 21, 2017 at 11:03 AM, Andy Christianson
<achristian...@hortonworks.com> wrote:
> Agree 100%. I have been bitten by this a few times. Is this something 
Aldrin can do/have done?
>
> -Andy I.C.
> 
> From: Jeff Zemerick <jzemer...@apache.org>
> Sent: Friday, August 18, 2017 2:56 PM
> To: dev@nifi.apache.org
> Subject: Separate MiNiFi projects in JIRA
>
> The MINIFI project in JIRA is currently a combination of issues for both
> the C++ and Java implementations. Some issues for the C++ project do have
> the C++ component set but some don't and it can sometimes be hard to 
easily
> differentiate the issues by their titles. (There isn't a "Java" component
> so a useful filter is hard to make.) Has there been any consideration 
given
> to having separate JIRA projects for the C++/Java MiNiFi implementations?
>
> Thanks,
> Jeff
>
>

Re: Separate MiNiFi projects in JIRA

[Andy Christianson]

Joe,

We actually already have that. There is a 'C++' and 'Java' component. It works 
for the most part, but there are cases where it becomes ambiguous, particularly 
on docker-related tickets.

I think there's certainly an argument that we need to just track components 
more carefully. Having it be a separate JIRA would make it harder to make a 
ticket ambiguous. Is it worth the effort/overhead of setting up another JIRA? 
I'll leave that to the more experienced/established Apache parties since I 
don't know what the overhead cost is.

Regards,

Andy I.C.

From: Joe Witt <joe.w...@gmail.com>
Sent: Monday, August 21, 2017 11:10 AM
To: dev@nifi.apache.org
Subject: Re: Separate MiNiFi projects in JIRA

Can we recommend and setup a set of component names so that filtering
can be done reasonably?

If we do that would it be sufficient?

Alternatively we can ask ASF infra to setup another JIRA project such
as 'minificpp' but I'd like to avoid that until we're really sure we
want to bug em.

Thanks

On Mon, Aug 21, 2017 at 11:03 AM, Andy Christianson
<achristian...@hortonworks.com> wrote:
> Agree 100%. I have been bitten by this a few times. Is this something Aldrin 
> can do/have done?
>
> -Andy I.C.
> 
> From: Jeff Zemerick <jzemer...@apache.org>
> Sent: Friday, August 18, 2017 2:56 PM
> To: dev@nifi.apache.org
> Subject: Separate MiNiFi projects in JIRA
>
> The MINIFI project in JIRA is currently a combination of issues for both
> the C++ and Java implementations. Some issues for the C++ project do have
> the C++ component set but some don't and it can sometimes be hard to easily
> differentiate the issues by their titles. (There isn't a "Java" component
> so a useful filter is hard to make.) Has there been any consideration given
> to having separate JIRA projects for the C++/Java MiNiFi implementations?
>
> Thanks,
> Jeff
>
>

Re: Separate MiNiFi projects in JIRA

[Andy Christianson]

Agree 100%. I have been bitten by this a few times. Is this something Aldrin 
can do/have done?

-Andy I.C.

From: Jeff Zemerick 
Sent: Friday, August 18, 2017 2:56 PM
To: dev@nifi.apache.org
Subject: Separate MiNiFi projects in JIRA

The MINIFI project in JIRA is currently a combination of issues for both
the C++ and Java implementations. Some issues for the C++ project do have
the C++ component set but some don't and it can sometimes be hard to easily
differentiate the issues by their titles. (There isn't a "Java" component
so a useful filter is hard to make.) Has there been any consideration given
to having separate JIRA projects for the C++/Java MiNiFi implementations?

Thanks,
Jeff

Re: Compilation error in minifi-cpp

[Andy Christianson]

OK. I created MINIFI-376 [1] to track the issue. I take it the fix is to remove 
the references? Do you recommend for this to be done in a bug fix branch, or 
will it be taken care of in one of your upcoming merges?

[1] https://issues.apache.org/jira/browse/MINIFI-376

Regards,

Andy I.C.

On 8/16/17, 10:36 AM, "Marc" <phroc...@apache.org> wrote:

Andy,
  In a branch I've been working on, I happened upon those  ( curlbuild.h
and easy.h which I believe are unnecessary ) , which were removed elsewhere
but apparently I didn't see them here due to copy/paste of some code and
removed them without issue.

On Wed, Aug 16, 2017 at 10:30 AM, Andy Christianson <
achristian...@hortonworks.com> wrote:

> MiNiFi cpp devs,
>
> Very recently I started getting this unexpected compilation error in the
> ‘make docker’ target. I double-checked the packages we are installing in
> the Dockerfile, and it does include curl-dev, so I’m confused why we’d be
> seeing this. Is it perhaps a curl version issue?
>
> [  3%] Performing build step for 'jsoncpp_project'
> Scanning dependencies of target jsoncpp_lib_static
> [ 25%] Building CXX object src/lib_json/CMakeFiles/
> jsoncpp_lib_static.dir/json_reader.cpp.o
> [ 50%] Building CXX object src/lib_json/CMakeFiles/
> jsoncpp_lib_static.dir/json_value.cpp.o
> [ 75%] Building CXX object src/lib_json/CMakeFiles/
> jsoncpp_lib_static.dir/json_writer.cpp.o
> [100%] Linking CXX static library libjsoncpp.a
> [100%] Built target jsoncpp_lib_static
> [  3%] Performing install step for 'jsoncpp_project'
> [100%] Built target jsoncpp_lib_static
> Install the project...
> -- Install configuration: "Release"
> -- Installing: /opt/minifi/build/jsoncpp/lib/pkgconfig/jsoncpp.pc
> -- Installing: /opt/minifi/build/jsoncpp/lib/libjsoncpp.a
> -- Installing: /opt/minifi/build/jsoncpp/include/json/allocator.h
> -- Installing: /opt/minifi/build/jsoncpp/include/json/assertions.h
> -- Installing: /opt/minifi/build/jsoncpp/include/json/autolink.h
> -- Installing: /opt/minifi/build/jsoncpp/include/json/config.h
> -- Installing: /opt/minifi/build/jsoncpp/include/json/features.h
> -- Installing: /opt/minifi/build/jsoncpp/include/json/forwards.h
> -- Installing: /opt/minifi/build/jsoncpp/include/json/json.h
> -- Installing: /opt/minifi/build/jsoncpp/include/json/reader.h
> -- Installing: /opt/minifi/build/jsoncpp/include/json/value.h
> -- Installing: /opt/minifi/build/jsoncpp/include/json/version.h
> -- Installing: /opt/minifi/build/jsoncpp/include/json/writer.h
> [  3%] Completed 'jsoncpp_project'
> [  3%] Built target jsoncpp_project
> Scanning dependencies of target minifi
> [  3%] Building CXX object libminifi/CMakeFiles/minifi.
> dir/src/core/logging/LoggerConfiguration.cpp.o
> [  4%] Building CXX object libminifi/CMakeFiles/minifi.
> dir/src/io/AtomicEntryStream.cpp.o
> [  4%] Building CXX object libminifi/CMakeFiles/minifi.
> dir/src/io/BaseStream.cpp.o
> [  5%] Building CXX object libminifi/CMakeFiles/minifi.
> dir/src/io/CRCStream.cpp.o
> [  5%] Building CXX object libminifi/CMakeFiles/minifi.
> dir/src/io/ClientSocket.cpp.o
> [  6%] Building CXX object libminifi/CMakeFiles/minifi.
> dir/src/io/DataStream.cpp.o
> [  6%] Building CXX object libminifi/CMakeFiles/minifi.
> dir/src/io/EndianCheck.cpp.o
> [  6%] Building CXX object libminifi/CMakeFiles/minifi.
> dir/src/io/FileStream.cpp.o
> [  7%] Building CXX object libminifi/CMakeFiles/minifi.
> dir/src/io/Serializable.cpp.o
> [  7%] Building CXX object libminifi/CMakeFiles/minifi.
> dir/src/io/StreamFactory.cpp.o
> [  8%] Building CXX object libminifi/CMakeFiles/minifi.
> dir/src/io/tls/TLSSocket.cpp.o
> [  8%] Building CXX object libminifi/CMakeFiles/minifi.
> dir/src/core/controller/ControllerServiceNode.cpp.o
> [  9%] Building CXX object libminifi/CMakeFiles/minifi.
> dir/src/core/controller/ControllerServiceProvider.cpp.o
> [  9%] Building CXX object libminifi/CMakeFiles/minifi.
> dir/src/core/controller/StandardControllerServiceNode.cpp.o
> [  9%] Building CXX object libminifi/CMakeFiles/minifi.
> dir/src/controllers/SSLContextService.cpp.o
> [ 10%] Building CXX object libminifi/CMakeFiles/minifi.
> dir/src/core/ClassLoader.cpp.o
> [ 10%] Building CXX object libminifi/CMakeFiles/minifi.dir/src/core/
> ConfigurableComponent.cpp.o
> [ 11%] Building CXX object libminifi/CMakeFiles/minifi.dir/src/core/
> ConfigurationFactory.cpp.o
> In file included from /opt/

Compilation error in minifi-cpp

[Andy Christianson]

MiNiFi cpp devs,

Very recently I started getting this unexpected compilation error in the ‘make 
docker’ target. I double-checked the packages we are installing in the 
Dockerfile, and it does include curl-dev, so I’m confused why we’d be seeing 
this. Is it perhaps a curl version issue?

[  3%] Performing build step for 'jsoncpp_project'
Scanning dependencies of target jsoncpp_lib_static
[ 25%] Building CXX object 
src/lib_json/CMakeFiles/jsoncpp_lib_static.dir/json_reader.cpp.o
[ 50%] Building CXX object 
src/lib_json/CMakeFiles/jsoncpp_lib_static.dir/json_value.cpp.o
[ 75%] Building CXX object 
src/lib_json/CMakeFiles/jsoncpp_lib_static.dir/json_writer.cpp.o
[100%] Linking CXX static library libjsoncpp.a
[100%] Built target jsoncpp_lib_static
[  3%] Performing install step for 'jsoncpp_project'
[100%] Built target jsoncpp_lib_static
Install the project...
-- Install configuration: "Release"
-- Installing: /opt/minifi/build/jsoncpp/lib/pkgconfig/jsoncpp.pc
-- Installing: /opt/minifi/build/jsoncpp/lib/libjsoncpp.a
-- Installing: /opt/minifi/build/jsoncpp/include/json/allocator.h
-- Installing: /opt/minifi/build/jsoncpp/include/json/assertions.h
-- Installing: /opt/minifi/build/jsoncpp/include/json/autolink.h
-- Installing: /opt/minifi/build/jsoncpp/include/json/config.h
-- Installing: /opt/minifi/build/jsoncpp/include/json/features.h
-- Installing: /opt/minifi/build/jsoncpp/include/json/forwards.h
-- Installing: /opt/minifi/build/jsoncpp/include/json/json.h
-- Installing: /opt/minifi/build/jsoncpp/include/json/reader.h
-- Installing: /opt/minifi/build/jsoncpp/include/json/value.h
-- Installing: /opt/minifi/build/jsoncpp/include/json/version.h
-- Installing: /opt/minifi/build/jsoncpp/include/json/writer.h
[  3%] Completed 'jsoncpp_project'
[  3%] Built target jsoncpp_project
Scanning dependencies of target minifi
[  3%] Building CXX object 
libminifi/CMakeFiles/minifi.dir/src/core/logging/LoggerConfiguration.cpp.o
[  4%] Building CXX object 
libminifi/CMakeFiles/minifi.dir/src/io/AtomicEntryStream.cpp.o
[  4%] Building CXX object 
libminifi/CMakeFiles/minifi.dir/src/io/BaseStream.cpp.o
[  5%] Building CXX object 
libminifi/CMakeFiles/minifi.dir/src/io/CRCStream.cpp.o
[  5%] Building CXX object 
libminifi/CMakeFiles/minifi.dir/src/io/ClientSocket.cpp.o
[  6%] Building CXX object 
libminifi/CMakeFiles/minifi.dir/src/io/DataStream.cpp.o
[  6%] Building CXX object 
libminifi/CMakeFiles/minifi.dir/src/io/EndianCheck.cpp.o
[  6%] Building CXX object 
libminifi/CMakeFiles/minifi.dir/src/io/FileStream.cpp.o
[  7%] Building CXX object 
libminifi/CMakeFiles/minifi.dir/src/io/Serializable.cpp.o
[  7%] Building CXX object 
libminifi/CMakeFiles/minifi.dir/src/io/StreamFactory.cpp.o
[  8%] Building CXX object 
libminifi/CMakeFiles/minifi.dir/src/io/tls/TLSSocket.cpp.o
[  8%] Building CXX object 
libminifi/CMakeFiles/minifi.dir/src/core/controller/ControllerServiceNode.cpp.o
[  9%] Building CXX object 
libminifi/CMakeFiles/minifi.dir/src/core/controller/ControllerServiceProvider.cpp.o
[  9%] Building CXX object 
libminifi/CMakeFiles/minifi.dir/src/core/controller/StandardControllerServiceNode.cpp.o
[  9%] Building CXX object 
libminifi/CMakeFiles/minifi.dir/src/controllers/SSLContextService.cpp.o
[ 10%] Building CXX object 
libminifi/CMakeFiles/minifi.dir/src/core/ClassLoader.cpp.o
[ 10%] Building CXX object 
libminifi/CMakeFiles/minifi.dir/src/core/ConfigurableComponent.cpp.o
[ 11%] Building CXX object 
libminifi/CMakeFiles/minifi.dir/src/core/ConfigurationFactory.cpp.o
In file included from 
/opt/minifi/libminifi/include/RemoteProcessorGroupPort.h:26:0,
 from /opt/minifi/libminifi/include/core/FlowConfiguration.h:23,
 from 
/opt/minifi/libminifi/include/core/ConfigurationFactory.h:22,
 from 
/opt/minifi/libminifi/src/core/ConfigurationFactory.cpp:25:
/opt/minifi/libminifi/include/utils/HTTPUtils.h:26:28: fatal error: 
curl/curlbuild.h: No such file or directory
 #include 
^
compilation terminated.
make[2]: *** [libminifi/CMakeFiles/minifi.dir/build.make:471: 
libminifi/CMakeFiles/minifi.dir/src/core/ConfigurationFactory.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:1884: libminifi/CMakeFiles/minifi.dir/all] 
Error 2
make: *** [Makefile:163: all] Error 2

Regards,

Andy I.C.

Re: MINIFI-350 minifi-cpp end-to-end integration testing framework

[Andy Christianson]

MiNiFi cpp team,

I have created the initial pytest/docker based test framework as well as a few 
initial test cases. Please review & merge the PR 
(https://github.com/apache/nifi-minifi-cpp/pull/126) at your convenience.

Regards,

Andy I.C.

From: kangax...@gmail.com <kangax...@gmail.com> on behalf of Haimo Liu 
<haimoli...@gmail.com>
Sent: Thursday, July 13, 2017 2:07 PM
To: dev@nifi.apache.org
Subject: Re: MINIFI-350 minifi-cpp end-to-end integration testing framework

great idea Andy! I can see this being extremely valuable even outside of
the MINIFI cpp context. Specifically, to migrate my dataflow from one
environment to another (DEV to QA to PROD), an integration testing
framework could be very helpful for flow validation purposes.

in addition to testing your MINIFI agents and network connectivities, have
you taken into consideration the integration testing of a potentially very
complex dataflow itself? Say I am collecting data from 50 data sources, and
ingesting to 20 different targets, may I leverage your testing framework to
spin up necessary containers (HDFS, Hbase, Oracle, etc. just different end
points) and run a docker compose script to validate my flow during
migration? Would be very nice to see your framework to be designed
extensible in a way to cover flow specific testing as well. Maybe you
already have it all sorted out :)

Thanks,
Haimo

On Thu, Jul 13, 2017 at 1:50 PM, Andy Christianson <
achristian...@hortonworks.com> wrote:

> Thanks for the feedback. I will put together a proof of concept which we
> can further evaluate/refine/merge upstream.
>
> -Andy
>
> On 7/13/17, 11:30 AM, "Kevin Doran" <kdoran.apa...@gmail.com> wrote:
>
> Great idea, Andy! Additional types of automated tests would help the
> minifi-cpp project significantly, and I think your proposal is an
> appropriate way to add integration tests for the minifi agent. This sounds
> like a great way to verify expected behavior of processors and the system
> of components in flow combinations.
>
> I like the idea of declarative tests that are interpreted / run by a
> harness or framework as a way to allow others can contribute test cases.
>
> I've never used the Bats framework before, but it seems like a
> reasonable option for what you describe. It might require writing a fair
> amount of bash code under-the-hood to get the functionality you want
> (helper functions and such), but it looks like it would keep the test cases
> themselves and the output clean and light. Perhaps others can offer
> suggestions here.
>
> One comment, which you've probably already considered, is that we
> should keep the dependencies (if any) that get added for integration tests
> that leverage the docker target optional so they are not required for folks
> that just want to build libminifi or the agent. It would be more of a
> developer/contributor option but users could skip these tests.
>
> /docker/test/integration seems like a reasonable place to add test
> cases. Others would probably know better. I think the README.md would be a
> reasonable place to document how to run the tests with a reference to
> another document that describes how to add / contribute new test cases. I'm
> not sure where the best location for the documentation should live.
>
> Thanks,
> Kevin
>
> On 7/13/17, 10:34, "Andy Christianson" <achristian...@hortonworks.com>
> wrote:
>
> Yes, I envision having a directory of declarative test cases. Each
> would include a flow yaml, one or more input files, and expected outputs.
>
> I’d like to document the convention before writing the
> implementation because if the conventions are solid, we can change out the
> actual test driver implementation later on if needed.
>
> Would it be best to document this in a section within /README.md,
> or should I add a new file such as /docs/Testing.md, or /TESTING.md?
>
> As for where the test cases would be added, I was thinking maybe
> /docker/test/integration, keeping consistent with the existing convention
> (i.e. /libminifi/test/integration).
>
> -Andy
>
> On 7/13/17, 10:14 AM, "Marc" <phroc...@apache.org> wrote:
>
> Hi Andy,
>I think this is a great idea to test integrating MiNiFi
> among multiple
> system components. Do you have a feel for how you will allow
> others to
> create test cases? Will you attempt to minimize the footprint
> of
> contributed tests by creating a bats based framework? I ask
> because it
> would be cool if contributors could supply a flow ( input )
> and expected
> output and

Re: MINIFI-350 CMake target for docker integration tests

[Andy Christianson]

These are good points. Having a separate target makes sense, plus it'll reduce 
risk of interfering with the existing native development workflow.

What shall we call the new target? Some possibilities:

- make sip (system integration tests)
- make docker-verify
- make verify-docker
- make integration

I'm open to ideas.

-Andy I.C.

From: Marc <phroc...@apache.org>
Sent: Wednesday, August 09, 2017 1:16 PM
To: dev@nifi.apache.org
Subject: Re: MINIFI-350 CMake target for docker integration tests

Andy,
  This is great stuff. To facilitate use by developers we should probably
come to agreement on terminology. In my opinion, it seems that you are
facilitating more system integration testing (SIT) based on our
discussions. While what you are doing doesn't preclude you from integrating
one or two components it does cause confusion on what to use when I want to
test something. I think the same exists for our unit and integration tests
that we have now, but those are a function of design decisions made vice a
focus on testability. I think most of the tests that use the docker
framework will be system tests, and the importance of this is that we
should probably isolate these to a separate target. I think better
isolation of what these test facilities do will end users, and limiting
what ctest executes to pure unit and modular integration tests will be very
useful.
   I think that also helps sell the story that one must install docker if
they want to run system integration tests, but you can still integrate
several components with ctest if you want to limit what you are testing. I
personally look forward to these changes as I think having multi host tests
will help find more bugs, but I wouldn't want this to run when I type make
test.

On Wed, Aug 9, 2017 at 12:39 PM, Andy Christianson <
achristian...@hortonworks.com> wrote:

> MiNiFi cpp team,
>
>
> I am currently working on MINIFI-350. I have an integration test framework
> plus a few integration tests which are invoked via a new script called
> docker/DockerVerify.sh (https://github.com/achristianson/nifi-minifi-cpp/
> blob/MINIFI-350/docker/DockerVerify.sh?). This is intended to be
> consistent in naming and structure to the extant DockerBuild.sh.
>
>
> My question resides in a final bit of integration. I see that there is a
> custom CMake target in cmake/DockerConfig.cmake that calls DockerBuild.sh.
> It seems like the best thing to integrate DockerVerify.sh is to add a
> sibling CMake custom target, perhaps called something like 'docker-verify.'
>
>
> A few stylistic/structural questions come to mind. Do we want these
> integration tests to be invoked directly via a custom target, or do we want
> that custom target to be an intermediate target which is called by some
> existing 'make test' or 'make verify' target? The biggest risk, from what I
> can see, is that if we hooked into 'make test,' then it would fail if the
> user hadn't already run 'make docker.'
>
>
> Please advise on preferred style and structure with regard to integration
> of the new docker integration tests with CMake.
>
>
> -Andy I.C.
>

MINIFI-350 CMake target for docker integration tests

[Andy Christianson]

MiNiFi cpp team,


I am currently working on MINIFI-350. I have an integration test framework plus 
a few integration tests which are invoked via a new script called 
docker/DockerVerify.sh 
(https://github.com/achristianson/nifi-minifi-cpp/blob/MINIFI-350/docker/DockerVerify.sh?).
 This is intended to be consistent in naming and structure to the extant 
DockerBuild.sh.


My question resides in a final bit of integration. I see that there is a custom 
CMake target in cmake/DockerConfig.cmake that calls DockerBuild.sh. It seems 
like the best thing to integrate DockerVerify.sh is to add a sibling CMake 
custom target, perhaps called something like 'docker-verify.'


A few stylistic/structural questions come to mind. Do we want these integration 
tests to be invoked directly via a custom target, or do we want that custom 
target to be an intermediate target which is called by some existing 'make 
test' or 'make verify' target? The biggest risk, from what I can see, is that 
if we hooked into 'make test,' then it would fail if the user hadn't already 
run 'make docker.'


Please advise on preferred style and structure with regard to integration of 
the new docker integration tests with CMake.


-Andy I.C.

MINIFI-368 exclude hidden files when scanning for src files

[Andy Christianson]

MiNiFi cpp team,


I have submitted a PR which fixes an issue where having a file open in vim 
causes the cmake build to fail. It fails because the BuildTests.cmake file 
includes hidden files in its scan for source files.


Please have a look at https://github.com/apache/nifi-minifi-cpp/pull/125? and 
merge in when possible.


-Andy C.

MINIFI-367 port tests to use boost::filesystem vs. stat.h for better portability

[Andy Christianson]

Hi All,

I submitted a PR to fix MINIFI-367. It uses boost::filesystem since this 
abstracts away POSIX feature detection and gives us better portability with 
less maintenance and developer time.

Please review & merge at your convenience.

-Andy C.

Re: [DISCUSS] Increasing durability in MiNiFi C++

[Andy Christianson]

In addition to the tickets mentioned, we probably want to do is isolate custom 
processors as much as possible. I.e. if a custom processor segfaults, we 
probably don’t want that to bring down the entire minifi process. Achieving 
that type of isolation might come with some tradeoffs, though. For instance, we 
may need to implement process-level isolation, similar to how the chromium 
browser isolates tab processes, but doing so would come with additional memory 
and IPC overhead. Maybe there are some modern sandboxing techniques we can look 
at.

Something to consider.

On 8/1/17, 9:59 AM, "Marc"  wrote:

Good Morning,

  I've begun capturing some details in a ticket for durability and
reliability of MiNiFi C++ clients [1]. The scope of this ticket is
continuing operations despite failure within specific components. There is
a linked ticket [2] attempts to address some of the concerns brought up in
MINIFI-356, focusing no memory usage.

  The spirit of the ticket was meant to capture conditions of known
failure; however, given that more discussion has blossomed, I'd like to
assess the experience of the mailing list. Continuing operations in any
environment is difficult, particularly one in which we likely have little
to no control. Simply gathering information to know when a failure is
occurring is a major part of the battle. According to the tickets, there
needs to be some discussion of how we classify failure.

  The ticket addressed the low hanging fruit, but there are certainly more
conditions of failure. If a disk switches to read/write mode, disks becomes
full and/or out of inode entries etc, we know a complete failure occurred
and thus can switch our type of write activity to use a volatile repo. I
recognize that partial failures may occur, but how do we classify these?
Should we classify these at all or would this be venturing into a rabbit
hole?

   For memory we can likely throttle queue sizes as needed. For networking
and other components we could likely find other measures of failure. The
goal, no matter the component, is to continue operations without human
intervention -- with the hope that the configuration makes the bounds of
the client obvious.

   My gut reaction is to separate partial failure as the low hanging fruit
of complete failure is much easier to address, but would love to hear the
reaction of this list. Further, any input on the types of failures to
address would be appreciated. Look forward to any and all responses.

  Best Regards,
  Marc

[1] https://issues.apache.org/jira/browse/MINIFI-356
[2] https://issues.apache.org/jira/browse/MINIFI-360

Re: minifi-cpp config.yml case-sensitivity

[Andy Christianson]

My idea of renaming the keys shouldn’t require any changes to yaml-cpp. The 
pseudocode would be (assuming a simplification that the whole tree is a map of 
maps, plus leaf values):

normalize(yamlTree):

  if is_leaf(yamlTree):
ret yamlTree

  newTree = {}

  for key in yamlTree: 
newKey = to_lower(key)
newTree[newKey] = normalize(yamlTree[key]) 

  ret newTree

So we would just normalize all key keys in the parsed yaml tree before mapping 
the yaml config values to minifi internal config structures.

-Andy

On 7/13/17, 1:53 PM, "Marc" <phroc...@apache.org> wrote:

Hey Andy,
The concern I had was not with performance -- as you said, only at load
and reload time -- but more with maintainability.

 If we make the change to the third party directory we'll have the
concern that we lose this capability when someone copies a new version of
yaml-cpp into our base directory, perhaps even at review. I'm not a fan of
maintaining someone else's code.

 Can we alleviate this concern by not making the change directly to
their code in the third party directory?. I briefly entertained the thought
of using a git submodule and custom branch, but there we would have the
same issue. My thought is can we do this in such a way that a copy/paste of
an updated version makes it glaringly obvious that there are changes here.
I don't think a section of the readme is likely to suffice.

On Thu, Jul 13, 2017 at 1:40 PM, Andy Christianson <
achristian...@hortonworks.com> wrote:

> On initial investigation, it doesn’t seem trivial to enable support for
> this in yaml-cpp. It might be feasible to walk the entire parsed yaml tree
> and map keys to a to_lower or to_upper key. It would come with some
> computational and possibly memory cost, but only at config load time, 
which
> shouldn’t be in the hot path. IMO, the usability improvement would be 
worth
> the cost.
>
> On 7/13/17, 11:16 AM, "Marc" <phroc...@apache.org> wrote:
>
> Hi Andy,
>   I believe this makes sense if Yaml-CPP supports the option, but I
> would
> be cautious if we have to change dependencies. If Yaml-CPP does not
> support
> this are you proposing to make the change to our third party 
directory?
>
> On Thu, Jul 13, 2017 at 11:09 AM, Andy Christianson <
> achristian...@hortonworks.com> wrote:
>
> > All,
> >
> > It appears that the properties in the config.yml are case-sensitive,
> and
> > to add to it, they’re inconsistent. E.g. in the example from the
> README.md
> > file, we have:
> >
> > …
> > Processors:
> > - name: GetFile
> >   id: 471deef6-2a6e-4a7d-912a-81cc17e3a206
> >   class: org.apache.nifi.processors.standard.GetFile
> >   max concurrent tasks: 1
> >   scheduling strategy: TIMER_DRIVEN
> >   scheduling period: 1 sec
> >   penalization period: 30 sec
> >   yield period: 1 sec
> >   run duration nanos: 0
> >   auto-terminated relationships list:
> >   Properties:
> >   Input Directory: /tmp/getfile
> >   Keep Source File: true
> > …
> >
> > The pragmatic course of action would seem to be making the config
> parser
> > case-insensitive, so that users have one less thing to worry about.
> >
> > If there is no opposition to this, I will enter a Jira ticket to
> make the
> > config parser case-insensitive.
> >
> > -Andy
> >
> >
>
>
>

Re: MINIFI-350 minifi-cpp end-to-end integration testing framework

[Andy Christianson]

Thanks for the feedback. I will put together a proof of concept which we can 
further evaluate/refine/merge upstream.

-Andy

On 7/13/17, 11:30 AM, "Kevin Doran" <kdoran.apa...@gmail.com> wrote:

Great idea, Andy! Additional types of automated tests would help the 
minifi-cpp project significantly, and I think your proposal is an appropriate 
way to add integration tests for the minifi agent. This sounds like a great way 
to verify expected behavior of processors and the system of components in flow 
combinations.

I like the idea of declarative tests that are interpreted / run by a 
harness or framework as a way to allow others can contribute test cases.

I've never used the Bats framework before, but it seems like a reasonable 
option for what you describe. It might require writing a fair amount of bash 
code under-the-hood to get the functionality you want (helper functions and 
such), but it looks like it would keep the test cases themselves and the output 
clean and light. Perhaps others can offer suggestions here.

One comment, which you've probably already considered, is that we should 
keep the dependencies (if any) that get added for integration tests that 
leverage the docker target optional so they are not required for folks that 
just want to build libminifi or the agent. It would be more of a 
developer/contributor option but users could skip these tests.

/docker/test/integration seems like a reasonable place to add test cases. 
Others would probably know better. I think the README.md would be a reasonable 
place to document how to run the tests with a reference to another document 
that describes how to add / contribute new test cases. I'm not sure where the 
best location for the documentation should live.

Thanks,
Kevin

On 7/13/17, 10:34, "Andy Christianson" <achristian...@hortonworks.com> 
wrote:

Yes, I envision having a directory of declarative test cases. Each 
would include a flow yaml, one or more input files, and expected outputs.

I’d like to document the convention before writing the implementation 
because if the conventions are solid, we can change out the actual test driver 
implementation later on if needed.

Would it be best to document this in a section within /README.md, or 
should I add a new file such as /docs/Testing.md, or /TESTING.md?

As for where the test cases would be added, I was thinking maybe 
/docker/test/integration, keeping consistent with the existing convention (i.e. 
/libminifi/test/integration).

-Andy

On 7/13/17, 10:14 AM, "Marc" <phroc...@apache.org> wrote:

Hi Andy,
   I think this is a great idea to test integrating MiNiFi among 
multiple
system components. Do you have a feel for how you will allow others 
to
create test cases? Will you attempt to minimize the footprint of
contributed tests by creating a bats based framework? I ask because 
it
would be cool if contributors could supply a flow ( input ) and 
expected
output and we automatically run the necessary 
containers/components. Is
this along the lines of your vision?

  Thanks,
   Marc
    
    On Wed, Jul 12, 2017 at 12:26 PM, Andy Christianson <
achristian...@hortonworks.com> wrote:

> Hi All,
>
> I am looking at MINIFI-350 and would like to implement some 
end-to-end
> integration tests for minifi cpp. Essentially, the tests would:
>
>
>   1.  Stand up a new minifi cpp docker container
>   2.  Send test data to HTTP input ports on the container
>   3.  Run data through a minifi flow
>   4.  Receive output data to a HTTP endpoint
>   5.  Verify output data according to some constraints (headers 
present,
> hash of the content, etc.)
>
> Most of this work, such as setting up a docker container and 
sending data
> to it, can naturally be done with shell commands. As such, I’ve 
taken a
> look at the bats [1] testing framework, which seems simple enough 
and is
> very expressive.
>
> Any thoughts or suggestions on test frameworks to use are 
appreciated.
>
> [1]: https://github.com/sstephenson/bats
>
>

Re: minifi-cpp config.yml case-sensitivity

[Andy Christianson]

On initial investigation, it doesn’t seem trivial to enable support for this in 
yaml-cpp. It might be feasible to walk the entire parsed yaml tree and map keys 
to a to_lower or to_upper key. It would come with some computational and 
possibly memory cost, but only at config load time, which shouldn’t be in the 
hot path. IMO, the usability improvement would be worth the cost.

On 7/13/17, 11:16 AM, "Marc" <phroc...@apache.org> wrote:

Hi Andy,
  I believe this makes sense if Yaml-CPP supports the option, but I would
be cautious if we have to change dependencies. If Yaml-CPP does not support
this are you proposing to make the change to our third party directory?

On Thu, Jul 13, 2017 at 11:09 AM, Andy Christianson <
achristian...@hortonworks.com> wrote:

> All,
>
> It appears that the properties in the config.yml are case-sensitive, and
> to add to it, they’re inconsistent. E.g. in the example from the README.md
> file, we have:
>
> …
> Processors:
> - name: GetFile
>   id: 471deef6-2a6e-4a7d-912a-81cc17e3a206
>   class: org.apache.nifi.processors.standard.GetFile
>   max concurrent tasks: 1
>   scheduling strategy: TIMER_DRIVEN
>   scheduling period: 1 sec
>   penalization period: 30 sec
>   yield period: 1 sec
>   run duration nanos: 0
>   auto-terminated relationships list:
>   Properties:
>   Input Directory: /tmp/getfile
>   Keep Source File: true
> …
>
> The pragmatic course of action would seem to be making the config parser
> case-insensitive, so that users have one less thing to worry about.
>
> If there is no opposition to this, I will enter a Jira ticket to make the
> config parser case-insensitive.
>
> -Andy
>
>

minifi-cpp config.yml case-sensitivity

[Andy Christianson]

All,

It appears that the properties in the config.yml are case-sensitive, and to add 
to it, they’re inconsistent. E.g. in the example from the README.md file, we 
have:

…
Processors:
- name: GetFile
  id: 471deef6-2a6e-4a7d-912a-81cc17e3a206
  class: org.apache.nifi.processors.standard.GetFile
  max concurrent tasks: 1
  scheduling strategy: TIMER_DRIVEN
  scheduling period: 1 sec
  penalization period: 30 sec
  yield period: 1 sec
  run duration nanos: 0
  auto-terminated relationships list:
  Properties:
  Input Directory: /tmp/getfile
  Keep Source File: true
…

The pragmatic course of action would seem to be making the config parser 
case-insensitive, so that users have one less thing to worry about.

If there is no opposition to this, I will enter a Jira ticket to make the 
config parser case-insensitive.

-Andy

Re: MINIFI-350 minifi-cpp end-to-end integration testing framework

[Andy Christianson]

Yes, I envision having a directory of declarative test cases. Each would 
include a flow yaml, one or more input files, and expected outputs.

I’d like to document the convention before writing the implementation because 
if the conventions are solid, we can change out the actual test driver 
implementation later on if needed.

Would it be best to document this in a section within /README.md, or should I 
add a new file such as /docs/Testing.md, or /TESTING.md?

As for where the test cases would be added, I was thinking maybe 
/docker/test/integration, keeping consistent with the existing convention (i.e. 
/libminifi/test/integration).

-Andy

On 7/13/17, 10:14 AM, "Marc" <phroc...@apache.org> wrote:

Hi Andy,
   I think this is a great idea to test integrating MiNiFi among multiple
system components. Do you have a feel for how you will allow others to
create test cases? Will you attempt to minimize the footprint of
contributed tests by creating a bats based framework? I ask because it
would be cool if contributors could supply a flow ( input ) and expected
output and we automatically run the necessary containers/components. Is
this along the lines of your vision?

  Thanks,
   Marc

On Wed, Jul 12, 2017 at 12:26 PM, Andy Christianson <
achristian...@hortonworks.com> wrote:

> Hi All,
>
> I am looking at MINIFI-350 and would like to implement some end-to-end
> integration tests for minifi cpp. Essentially, the tests would:
>
>
>   1.  Stand up a new minifi cpp docker container
>   2.  Send test data to HTTP input ports on the container
>   3.  Run data through a minifi flow
>   4.  Receive output data to a HTTP endpoint
>   5.  Verify output data according to some constraints (headers present,
> hash of the content, etc.)
>
> Most of this work, such as setting up a docker container and sending data
> to it, can naturally be done with shell commands. As such, I’ve taken a
> look at the bats [1] testing framework, which seems simple enough and is
> very expressive.
>
> Any thoughts or suggestions on test frameworks to use are appreciated.
>
> [1]: https://github.com/sstephenson/bats
>
>

MINIFI-350 minifi-cpp end-to-end integration testing framework

[Andy Christianson]

Hi All,

I am looking at MINIFI-350 and would like to implement some end-to-end 
integration tests for minifi cpp. Essentially, the tests would:


  1.  Stand up a new minifi cpp docker container
  2.  Send test data to HTTP input ports on the container
  3.  Run data through a minifi flow
  4.  Receive output data to a HTTP endpoint
  5.  Verify output data according to some constraints (headers present, hash 
of the content, etc.)

Most of this work, such as setting up a docker container and sending data to 
it, can naturally be done with shell commands. As such, I’ve taken a look at 
the bats [1] testing framework, which seems simple enough and is very 
expressive.

Any thoughts or suggestions on test frameworks to use are appreciated.

[1]: https://github.com/sstephenson/bats