Hi, no errors can be detected on the sender side (even the nic will not detect if the other side is down or not). If a UDP packet is lost, the receiver side will detect and log it. There are diodes that are using an archive. If any transmission is lost you can easily resend the data manually.
Practically one proceeds as follows: - know your hardware very (very) well. Know how many packets can be sent without loss (that is most important). - there is only one process that sends data; No overload (congestion) may be generated - counters are used to detect packet loss (some diodes use something like ecc) I know diodes that transfer millions of data without any loss. It is not so difficult as it sounds. But of course there is always the possibility of a packet loss without recognizing it immediately. It is always a tradeoff between security and convenience. Using two diodes always means having two separate Nifi systems. We do generate a unique ID that is always part of the flow (like the uuid). This ID is transferred across all systems. All logs contain this ID. So you see the flow across the network border. We use nifi to synchronize a lot of data across separated networks. And I don't know of any system that can do this better. Regards Marc > Am 01.08.2021 um 23:15 schrieb Phil H <gippyp...@gmail.com>: > > That is interesting stuff - out of interest, if it was sent over that UDP > diode, how would you know whether or not it got to the other side? I > haven’t looked into the site-to-site functionality much yet but I assume it > maintains the providence info? > > On Mon, 2 Aug 2021 at 04:26, Marc <n...@nerdfunk.net > <mailto:n...@nerdfunk.net>> wrote: > >> Greetings, >> >> there are companies and organizations that strictly separate their >> networks for security reasons. Such companies often use diodes to achieve >> this. But of course they still have to exchange data between the networks >> (eg. transfer data from ‚low‘ to ‚high‘). There are at least two kinds of >> diodes. Some hardware-based ones only use one fiber optic to send data (UDP >> based). Others use TCP, but prevent sending in the reverse direction. >> >> Nifi is an amazing tool that allows data to be transferred between two >> separate networks in a very flexible but also secure way. I have >> implemented two processors. The first one ‚merges‘ the attributes and the >> content of a flowfile and sends it to the destination. The second one >> listens on a TCP port, splits attributes and content and creates a new >> flowfile containing all attributes of the origin flow. You can send the >> flow without attributes as well. In this case you can easily netcat a >> binary file to Nifi. >> >> These two processors are useful if you do NOT have a bidirectional >> communication between two NiFi instances and therefore the site-2-site >> mechanism or http(s) cannot be used. >> >> We have been using these processors for a longer period of time (exactly >> the version for 1.13.2) and would like to share these processors with >> others. So the question to you all is: Is someone interested in these >> processors or is this use case too special? >> >> The current source code can be found on GitHub. ( >> https://github.com/nerdfunk-net/diode/ >> <https://github.com/nerdfunk-net/diode/> < >> https://github.com/nerdfunk-net/diode/ >> <https://github.com/nerdfunk-net/diode/>>) >> >> I have also implemented a UDP based version of the processor. Due to the >> nature of UDP, this is more complex and these processors are now being >> tested. >> >> Best regards >> Marc
