Re: Using nifi in separated networks

[Phil H]

That is interesting stuff - out of interest, if it was sent over that UDP
diode, how would you know whether or not it got to the other side? I
haven’t looked into the site-to-site functionality much yet but I assume it
maintains the providence info?

On Mon, 2 Aug 2021 at 04:26, Marc  wrote:

> Greetings,
>
> there are companies and organizations that strictly separate their
> networks for security reasons. Such companies often use diodes to achieve
> this. But of course they still have to exchange data between the networks
> (eg. transfer data from ‚low‘ to ‚high‘). There are at least two kinds of
> diodes. Some hardware-based ones only use one fiber optic to send data (UDP
> based). Others use TCP, but prevent sending in the reverse direction.
>
> Nifi is an amazing tool that allows data to be transferred between two
> separate networks in a very flexible but also secure way. I have
> implemented two processors. The first one ‚merges‘ the attributes and the
> content of a flowfile and sends it to the destination. The second one
> listens on a TCP port, splits attributes and content and creates a new
> flowfile containing all attributes of the origin flow. You can send the
> flow without attributes as well. In this case you can easily netcat a
> binary file to Nifi.
>
> These two processors are useful if you do NOT have a bidirectional
> communication between two NiFi instances and therefore the site-2-site
> mechanism or http(s) cannot be used.
>
> We have been using these processors for a longer period of time (exactly
> the version for 1.13.2) and would like to share these processors with
> others. So the question to you all is: Is someone interested in these
> processors or is this use case too special?
>
> The current source code can be found on GitHub. (
> https://github.com/nerdfunk-net/diode/ <
> https://github.com/nerdfunk-net/diode/>)
>
> I have also implemented a UDP based version of the processor. Due to the
> nature of UDP, this is more complex and these processors are now being
> tested.
>
> Best regards
> Marc

Using nifi in separated networks

[Marc]

Greetings,

there are companies and organizations that strictly separate their networks for 
security reasons. Such companies often use diodes to achieve this. But of 
course they still have to exchange data between the networks (eg. transfer data 
from ‚low‘ to ‚high‘). There are at least two kinds of diodes. Some 
hardware-based ones only use one fiber optic to send data (UDP based). Others 
use TCP, but prevent sending in the reverse direction.

Nifi is an amazing tool that allows data to be transferred between two separate 
networks in a very flexible but also secure way. I have implemented two 
processors. The first one ‚merges‘ the attributes and the content of a flowfile 
and sends it to the destination. The second one listens on a TCP port, splits 
attributes and content and creates a new flowfile containing all attributes of 
the origin flow. You can send the flow without attributes as well. In this case 
you can easily netcat a binary file to Nifi.

These two processors are useful if you do NOT have a bidirectional 
communication between two NiFi instances and therefore the site-2-site 
mechanism or http(s) cannot be used.

We have been using these processors for a longer period of time (exactly the 
version for 1.13.2) and would like to share these processors with others. So 
the question to you all is: Is someone interested in these processors or is 
this use case too special?

The current source code can be found on GitHub. 
(https://github.com/nerdfunk-net/diode/ 
)

I have also implemented a UDP based version of the processor. Due to the nature 
of UDP, this is more complex and these processors are now being tested.

Best regards
Marc

Re: [DISCUSS] NiFi 2.0 Release Goals

[Mark Bean]

I created a JIRA ticket to investigate and improve the performance of
InvokeHTTP. It includes a flow definition for benchmarking the performance
of both PostHTTP and InvokeHTTP.

https://issues.apache.org/jira/browse/NIFI-8968

Thanks,
Mark

On Fri, Jul 30, 2021 at 6:41 PM Adam Taft  wrote:

> I'm not seeing the side thread that was going to discuss deprecation of
> PostHTTP.  Has that thread started and I just don't see it?
>
> One (significant?) concern with PostHTTP is the smooth integration of
> NiFi-to-NiFi communication that is very transparently enabled with the
> ListenHTTP and PostHTTP processors. There's some special logic in there for
> handling flowfiles that InvokeHTTP doesn't really (nor should really) have.
>
> I know of several (large) NiFi installations that rely on the PostHTTP /
> ListenHTTP combination. It has enabled NiFi to NiFi communication for folks
> reluctant or unable to enable site-to-site type configuration.
>
> Honestly, I don't know why we'd want to "deprecate" any processor, as
> opposed to just moving it to a new location. If these processors can be
> ported and maintained to whatever the 2.0 API looks like, there's possibly
> little harm keeping them around.
>
> And by the way, what happened to the "marketplace" concept? Is this being
> considered for 2.0 as well?  Because relocating the deprecated processors
> to an external nar might be the best solution. Losing PostHTTP entirely I
> think would be a mistake, but I'd conceptually support relocating it.
>
> Thanks,
>
> /Adam
>
> On Tue, Jul 27, 2021 at 2:11 PM Joe Witt  wrote:
>
> > Looks like we just need to knock out 5 JIRAs :) [1]
> >
> > I felt like we had a label folks were using at one point but quickly
> > looking revealed nothing exciting.  After this confluence page
> > stabilizes a bit we can probably knock out some JIRAs and such.
> >
> > [1] https://issues.apache.org/jira/projects/NIFI/versions/12339599
> >
> > On Tue, Jul 27, 2021 at 1:06 PM Otto Fowler 
> > wrote:
> > >
> > >  I find myself wishing I had a list of all the jiras / issues that have
> > > been put off for a 2.0 release because they required some change or
> > another
> > > :(
> > >
> > > From: Joe Witt  
> > > Reply: dev@nifi.apache.org  
> > > Date: July 27, 2021 at 12:30:35
> > > To: dev@nifi.apache.org  
> > > Subject:  Re: [DISCUSS] NiFi 2.0 Release Goals
> > >
> > > A few thoughts:
> > >
> > > 1. I would love to see deprecation notices show up in the UI in
> > > various ways to help motivate users to move off things to more
> > > supportable things. That is not a prerequisite for anything happening
> > > however. Just a good feature/nice thing to do for users when someone
> > > is able to tackle it.
> > >
> > > 2. The decision to deprecate something and to further remove it need
> > > not mean there is a superior solution available. If that thing itself
> > > isn't getting the love/attention it needs to be
> > > maintained/supported/healthy going forward that alone is enough to
> > > remove it. That might well be the case with PostHTTP [1] and for
> > > comparison you can see how much effort has gone into InvokeHTTP [2].
> > >
> > > 3. When discussing a 2.0 release each thing we add as a 'must do' the
> > > further away from reality such a release will become. We'll have to
> > > get very specific about 'musts' vs 'wants'.
> > >
> > > [1]
> > >
> >
> https://github.com/apache/nifi/commits/11e9ff377333784974fa55f41483c4281d80da50/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PostHTTP.java
> > > [2]
> > >
> >
> https://github.com/apache/nifi/commits/cc554a6b110dfa45767bcb13d834ea4265d6dfe6/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java
> > >
> > > On Tue, Jul 27, 2021 at 8:47 AM David Handermann
> > >  wrote:
> > > >
> > > > Thanks Mark, providing a template or comparison statistics with Java
> > > > versions and component configuration details would be very helpful.
> If
> > it
> > > > is possible to run tests using a public API or deployable service,
> that
> > > > would also help confirm potential differences.
> > > >
> > > > Showing a deprecation notice in the UI could be helpful, perhaps as a
> > > > configurable option. NIFI-8650 describes a general Flow Analysis
> > > > capability, and it seems like that might be one possible way to
> surface
> > > > deprecation warnings. For something more specific to component
> > > deprecation,
> > > > it seems important to find a balance between making it obvious and
> > making
> > > > it something that ends up getting ignored.
> > > >
> > > > Regards,
> > > > David Handermann
> > > >
> > > > On Tue, Jul 27, 2021 at 10:28 AM Mark Bean 
> > wrote:
> > > >
> > > > > I'll start a new thread for PostHTTP when I get a template and/or
> > > detailed
> > > > > stats.
> > > > >
> > > > > I know the deprecation is noted in the documentation. That's a
> > > ne