Re: [DISCUSS] Finalizing Release Goals for NiFi 2.0

[ski n]

I read the proposal and seems like a very clear plan. The only thing I
noticed:

"Spring 6 
removed support for Java 8 in November 2022 "

On the release notes of Spring Framework 6.0 it says:

" As a major revision of the core framework, Spring Framework 6.0 comes
with a Java 17+ baseline and a move to Jakarta EE 9+ "

When JDK 11 is the baseline for NiFi, can it use Spring Framework 6.0?

And on Jakarta EE 9+, I see a lot of projects now switching from Javax to
Jakarta (for example Camel:
https://issues.apache.org/jira/browse/CAMEL-14680)

What will NiFi do with Jakarta namespace?

Raymond




On Wed, Dec 7, 2022 at 8:20 PM Mark Bean  wrote:

> I agree this is a great start to a discussion with pointers to important
> docs for the 2.0 transition. Thanks David!
>
> Mike - what do you mean by "controller service-based configuration for
> connection details"?
>
> Also, the transition from Java 11 to 17 is not without potential issues.
> I've discovered one already. [1] I support stepping up on Java version
> requirements. Perhaps rather than the currently stated "Requires Java 8 or
> Java 11", the requirement can be "Requires Java 11 or Java 17". I don't
> think you were suggesting the minimum version be Java 17, were you? Either
> way, the issue with Java 17 needs to be identified and fixed as well as
> more thorough testing to find other possible edge cases before we move
> forward too aggressively.
>
> [1] https://issues.apache.org/jira/browse/NIFI-10958
>
> On Wed, Dec 7, 2022 at 1:33 PM Mike Thomsen 
> wrote:
>
> > Really good start on the discussion. One thing I'm curious about is
> > Java 11 vs 17. Java 8 -> 11 is major jump that I can understand why
> > businesses scoffed at that for a long time, but the lift from 11 to 17
> > was about like 7 -> 8. A 2.0 release seems like a good time to jump
> > straight to the latest official LTS for Java and start greenlighting
> > new language features that might simplify things.
> >
> > I would also add (since I didn't see it) a design goal of forcing a
> > complete shift in all bundles to using controller service-based
> > configurations for connection details. 2.0 feels like a really good
> > time for us to establish a community-wide best practice of
> > centralizing configurations in dedicated components.
> >
> > On Wed, Dec 7, 2022 at 9:13 AM Mark Payne  wrote:
> > >
> > > Yeah, agreed. I am very supportive, as well.
> > >
> > > Thanks for taking the time to put this together, David.
> > >
> > > -Mark
> > >
> > >
> > > > On Dec 7, 2022, at 4:07 AM, Pierre Villard <
> > pierre.villard...@gmail.com> wrote:
> > > >
> > > > Thanks for putting this together David. This is an excellent writeup
> > and
> > > > it's great to have a release where we focus on tech debt as well as
> > making
> > > > sure we stay up to date with our dependencies and what we support.
> > This is
> > > > a great opportunity for us to clean a lot of things in our code and I
> > can't
> > > > wait for us to get started with this. I'm definitely a +1 to have a
> > formal
> > > > vote on this proposal.
> > > >
> > > > Thanks,
> > > > Pierre
> > > >
> > > > Le mar. 6 déc. 2022 à 23:50, Joe Witt  a écrit :
> > > >
> > > >> David, All,
> > > >>
> > > >> This is an excellent writeup/good framing.  I am supportive of this
> > > >> as-is since it is achievable and lays out a clear path.  We can make
> > > >> milestone releases of NiFi 2.0.0 along the way until we achieve all
> > > >> the stated goals. I assume migration bits will be the long pole and
> > > >> once we have them sorted we can kick out a 2.0.0.   We already have
> a
> > > >> version guide that governs how long we'd keep 1.x maintained though
> > > >> the phase out is pretty natural as we move main to a 2.0.0 basis
> > > >> anyway.
> > > >>
> > > >> Not to confuse this thread but it makes me think we could do a
> similar
> > > >> framing for a NiFi 3.0 which lays out a potentially new approach to
> > > >> NiFi decoupling the web/interface from the runtime/operations and
> one
> > > >> which is more fundamentally K8S based.  But we can cross that
> bridge a
> > > >> bit later.  Does seem more and more like folks in the community
> would
> > > >> like to know more about the potential directions we can go.
> > > >>
> > > >> Thanks!
> > > >> Joe
> > > >>
> > > >> On Tue, Dec 6, 2022 at 1:53 PM David Handermann
> > > >>  wrote:
> > > >>>
> > > >>> Team,
> > > >>>
> > > >>> With the release of NiFi 1.19.0 deprecating support for Java 8, the
> > end
> > > >> of
> > > >>> the year provides a good opportunity for finalizing general release
> > goals
> > > >>> for NiFi 2.0.
> > > >>>
> > > >>> Based on previous discussions from July 2021 [1] and June 2022 [2],
> > there
> > > >>> seems to be general agreement with focusing a NiFi 2.0 release on
> > > >> reducing
> > > >>> technical debt while providing a straightforward upgrade path for
> > current
> > > >>> deployments.
> > > 

Re: understand nifi usage patterns

[ski n]

"The use cases for which Nifi is used/adopted does the data load belongs to
streaming mode or pure batch workloads"

Actually for both. NiFi is used for creating data flows.

Data flows can be used for:

- Data ingesting (gathering data from a lot of sources)
- Data offloading (load data to other platforms like Apache Kafka, Hadoop,
Azure Data Explorer etc)
- Data integration (Connecting applications)
- Edge data (gathering data from IOT sensors) / in combination with MiNiFi)

Besides such use cases Apache NiFi does a good job in visualize and manage
these data flows and scale them when needed.

More reading: https://nifi.apache.org/docs/nifi-docs/html/overview.html

Raymond






On Thu, Dec 1, 2022 at 12:19 PM Tanmaya Panda
 wrote:

> Hi Team,
>
> We are working on developing a Nifi connector to send and receive data
> from Azure Data Explorer( a fast and fully managed data analytics service
> offering for relatime analysis on large volumes of data streaming from IOT
> devices, applications etc.).
>
> We wanted to understand the general usage patterns of Nifi. The use cases
> for which Nifi is used/adopted does the data load belongs to streaming mode
> or pure batch workloads?
>
> Thanks,
> Tanmaya
>

Re: [RESULT][VOTE] Release Apache NiFi 1.13.0

[ski n]

Small thing. The release date on "http://nifi.apache.org/download.html;
still says: Released September 28, 2020

Regards,

Raymond

Op ma 15 feb. 2021 om 18:16 schreef Otto Fowler :

> Great job everyone, thanks Joe!
>
> > On Feb 15, 2021, at 11:48, Joe Witt  wrote:
> >
> > Apache NiFi Community,
> >
> > I am pleased to announce that the 1.13.0 release of Apache NiFi passes
> with
> >6 +1 (binding) votes
> >7 +1 (non-binding) votes
> >0 0 votes
> >0 -1 votes
> >
> > Thanks to all who helped make this release possible. Several RCs many of
> > you stuck with and some new voters.  Great stuff!
> >
> > Here is the PMC vote thread:
> >
> https://lists.apache.org/thread.html/r7f8e56e60a9f2da7eb09da7ba696142a81c9dd19e374fca25a536cd1%40%3Cdev.nifi.apache.org%3E
> >
> > On Mon, Feb 15, 2021 at 9:43 AM Joe Witt  wrote:
> >
> >> +1 binding
> >>
> >> On Sun, Feb 14, 2021 at 3:26 AM Pierre Villard <
> >> pierre.villard...@gmail.com> wrote:
> >>
> >>> +1 binding
> >>>
> >>> Went through the usual verification process. Deployed secured cluster
> on
> >>> Google Cloud and checked integration with the NiFi Registry.
> >>>
> >>> Thanks,
> >>> Pierre
> >>>
> >>> Le dim. 14 févr. 2021 à 04:45, Peter Turcsanyi 
> a
> >>> écrit :
> >>>
>  +1 non-binding
> 
>  Went through the release helper guide.
>  Verified full build with empty local maven repo with Java 8
> >>> (AdoptOpenJDK
>  1.8.0_282-b08) and 11 (AdoptOpenJDK 11.0.10+9) on Ubuntu 20.04.
>  Ran full builds with different time zone settings to verify NIFI-8023,
> >>> also
>  tested it with flows.
>  Ran some SSL related test flows with processors changed recently
>  (ListenHTTP, GRPC, MQTT).
>  Verified --wait-for-init on Ubuntu and CentOS.
>  Verified NiFi web server listening on localhost only by default.
> 
>  Thanks,
>  Peter
> 
>  On Sun, Feb 14, 2021 at 12:56 AM David Handermann <
>  exceptionfact...@gmail.com> wrote:
> 
> > +1 non-binding
> >
> > Verified release signatures and expected files.
> > Verified build on Ubuntu 20.10 using Apache Maven 3.6.3 with Azul
> Zulu
>  JDK
> > 11.0.10 and AdoptOpenJDK 1.8.0_282.
> > Verified absence of startup script shell warnings on Ubuntu 20.10.
> > Verified service listening on localhost when using default
>  nifi.properties.
> > Configured and tested mutual TLS authentication on a standalone
> server
>  with
> > BCFKS key store and trust store.
> >
> > Regards,
> > David Handermann
> >
> > On Fri, Feb 12, 2021 at 5:45 PM Muazma Zahid 
> >>> wrote:
> >
> >> +1 (non-binding)
> >>
> >> - Ran build with OpenJDK 1.8.0_275 on Linux
> >> - Deployed cluster on Azure and tested flows with Blob, ADLS, and
> > CosmosDB
> >> processors.
> >>
> >> Looks good to me.
> >>
> >> Thanks
> >> Muazma
> >>
> >> On Fri, Feb 12, 2021 at 3:38 PM Sushil Kumar 
>  wrote:
> >>
> >>> +1 (non-binding) Release this package as nifi-1.13.0
> >>>
> >>> Deployed this via helm chart(
> >>> https://github.com/sushilkm/nifi-chart)
> > on
> >>> kubernetes.
> >>> Thank you to all the contributors and reviewers.
> >>>
> >>> On Fri, Feb 12, 2021 at 2:13 PM Marc Parisi 
> > wrote:
> >>>
>  +1
>  - Verified sigs and hashes
>  - Built on PopOS w/ java 8 && 11
>  - Tested with basic flow sending data to various devices w/
> >>> Secured
>  transport
>  - Tested secured w/ secured nifi reg.
> 
>  Thanks,
>  Marc
> 
>  On Fri, Feb 12, 2021 at 2:56 PM Andrew Lim <
> > andrewlim.apa...@gmail.com
> >>>
>  wrote:
> 
> > +1 (binding)
> >
> > -Tested secure NiFi with secure NiFi Registry 0.8.0
> > -Ran basic flows successfully
> > -Tested basic versioned flow functionality
> > -Reviewed and tested Core UI fixes and Documentation updates
> >
> > Drew
> >
> >> On Feb 10, 2021, at 11:37 PM, Joe Witt 
> > wrote:
> >>
> >> Hello,
> >>
> >> I am pleased to be calling this vote for the source release
> >>> of
> >> Apache
> > NiFi
> >> 1.13.0.
> >>
> >> The source zip, including signatures, digests, etc. can be
>  found
> >> at:
> >>
> >>>
>  https://repository.apache.org/content/repositories/orgapachenifi-1178
> >>
> >> The source being voted upon and the convenience binaries
> >>> can be
> >> found
>  at:
> >> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.13.0/
> >>
> >> A helpful reminder on how the release candidate verification
> >> process
> > works:
> >>
> >
> 
> >>>
> >>
> >
> 
> >>>
> 

Re: Biztalk VS. Nifi

[ski n]

Hi Jim,

There is extensive documentation here:

http://nifi.apache.org/docs.html

But I'll agree with Mike it's best to start with use cases. Are there use
cases in current BizTalk that you want to keep and are there also new use
cases. These use cases can be turned into real-world cases where you can
make a proof-of-concept with a simple one node NiFi setup. Often there are
commercial parties in your country which can assist you.

Raymond





Op wo 7 okt. 2020 om 20:12 schreef Mike Thomsen :

> Jim,
>
> It would be helpful to understand what your use cases are with BizTalk
> first. It's always difficult to compare products that are in similar,
> but not same, categories like rules engines vs data orchestration
> tools.
>
> I think as a starting point, NiFi is probably the world leader at this
> point in terms of options for integration between the official
> convenience binaries and what the community has published.
>
> Thanks,
>
> Mike
>
> On Wed, Oct 7, 2020 at 1:53 PM Jim Weier 
> wrote:
> >
> > Hey all
> >
> > I work for a company that uses Biztalk extensively for Dataflow, ETL,
> Application integration
> >
> > We are looking a Nifi as a Business rule engine,
> > I am looking for supporting documentation that shows this would be a
> good idea, VS using Biztalk Business rule engine
> >
> >
> > Any direction on this would be great
> > 
> >
> > If you're reading this, it's probably because you pay attention to
> detail. Well, one detail we never overlook at Veterans United Home Loans is
> serving our nation's Veterans, service members, and military families. This
> dedication has resulted in a 98% customer recommendation rating from the
> more than 200,000 customer reviews <
> https://www.veteransunited.com/reviews/?utm_source=email_medium=signature>
> we've received since 2013.
> >
> > NOTICE: Email is not a secure medium. If you have important documents
> for your loan team you can securely upload them to your account at
> my.veteransunited.com or provide this
> information by fax, mail, or phone. Please don't send sensitive personal
> information regarding your loan or personal identity in your emails or as
> an attachment.
> >
> > Mortgage Research Center, LLC is an Equal Opportunity Lender, not
> endorsed or affiliated with a government agency. NMLS # 1907 (
> nmlsconsumeraccess.org). Licensed in all
> 50 states. For State Licensing information, please visit
> www.veteransunited.com/licenses/.
>

Camel & Nifi

[ski n]

 Wrote a tech blog on using Camel with Apache NiFi:

https://medium.com/@raymondmeester/using-camel-and-nifi-in-one-solution-c7668fafe451

I thought I share it here,

Raymond

Re: Support for Java 14

[ski n]

These are the Long Term Support version:

Java 8
Java 11
Java 17 (planned September 21)

See: https://en.wikipedia.org/wiki/Java_version_history

Op zo 12 apr. 2020 om 22:44 schreef Mike Thomsen :

> Ok. I thought 14 was one. I may see what basic issues we can run down
> because it dies on startup because of a version checking issue in RunNiFi
> that will probably bite us at that point.
>
> On Sun, Apr 12, 2020 at 4:29 PM Joe Witt  wrote:
>
> > Mike
> >
> > Nope.  Dont think we would bother much unless its an LTS release.  Java
> 17
> > appears to be the next one and is scheduled for Sep ‘21
> >
> > thanks
> >
> > On Sun, Apr 12, 2020 at 3:51 PM Mike Thomsen 
> > wrote:
> >
> > > Has anyone started looking into this yet? I started playing around with
> > > running on adoptopenjdk 14 and ran into some problems right away on
> > startup
> > > and might have some time to play around with this if no else has
> started.
> > >
> > > Thanks,
> > >
> > > Mike
> > >
> >
>

Asset Registry

[ski n]

The NiFi Registry initially targeted at versioning flows at first. As its
goal is  much broader, namely as a complementary application that provides
a central location for storage and management of shared resources across
one or more instances of NiFi and/or MiNiFi, there have been two feature
proposals:

1) Variable registry (
https://cwiki.apache.org/confluence/display/NIFI/Variable+Registry)
2) Extension registry (
https://cwiki.apache.org/confluence/display/NIFI/Extension+Repositories+%28aka+Extension+Registry%29+for+Dynamically-loaded+Extensions
)

The first one is completed (NiFi 0.5) and the second one is on going.


Asset registry

The have a complete registry I want to discuss a fourth registry type: the
"Asset Registry". (As already shortly discussed with bbende on Slack).The
goal of Asset Registry would be to central store, versioning and manage
file resources.

Some examples would be:

1) jar libraries for connecting with MQ
2) XSLT files used in XML Transform processor
3) Configuration files
etc

So basically all kind of files that are used from/within NiFi.

Process

The process would be that one commit of a flow/process group that the files
linked to flow are committed to the Asset Repository with a link to flows
where a file is used. On import of flows the files are imported as file.
Probably the files are stored relatively to the NiFi Installation Dir, or
elsewhere when configured differently. This process should also work for
parameterized flows.

For example in my flow (On my Windows laptop):

1) Hard coded file link: C:\directory\myFile.ext
2) Parameterized file link: ${myPath}/myFile.ext

This should still work if for example

1) Another developer imports this on his MacBook
2) This is imported on Windows Server NiFi Installation
3) On a NiFi running on a Docker with Linux
etc

Besides committing/exporting/importing from NiFi this could also be done
from registry API, CLI or the Registry GUI. I think this would be a great
enhancement as this would provide one-stop solution for all kind of
resources.

What do you think?

Regards,

Raymond

Dutch NiFi User Group meetup

[ski n]

Hi All,

On the 15th of January 2020 I organize the Dutch NiFi User Group meetup in
Utrecht. Just an evening to talk about NiFi and to exchange technical
experiences. For those interested and based in the Netherlands, you can
registrar via this link:

https://www.eventbrite.nl/e/tickets-dutch-nifi-user-group-85081692633


Note: This is a free/open meetup targeted at NiFi users/developers only

Regards,

Raymond

Re: Contribution

[ski n]

Currently, there is no community repository (maybe part of NiFi Registry in
the future). On Github there is list of custom
https://github.com/jfrazee/awesome-nifi (But this seems not actively
maintained).

In case you want to improve a processor you can make a patch or pull
request as described here:
https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide

Regards,

Raymond

Op do 17 okt. 2019 om 20:17 schreef AbdelWahid BENZERROUK <
benzerrouk.wa...@gmail.com>:

> Hello,
>
> I have created a NiFi custom processor, and I wish to contribute with it ,
> to be an  alternative of PutIgniteCache  .
>
>
> Thank you,
>

Re: NIfi relace text for XML flow file content

[ski n]

I would suggest to use XSLT with the TransformXML Processor (
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.2/org.apache.nifi.processors.standard.TransformXml/
)

It's the most precise way to work with XML (better than for example
ReplaceText with regular expressions). There enough examples to find to do
this task with XSLT.

Regards,

Raymond


Op vr 20 sep. 2019 om 06:22 schreef Puspak Das :

> Hi team ,
>
> I have a xml flow file content where an particular tag is under there
> inside several parent tag .
> but i just want to replace one particular tag value with an attribute
> value of the same flow file .
>
> Please suggest how to achive it .
>
> Regards
> Puspak
>
>
>

Re: [EXT] Re: NiFi 2.0 Roadmap

[ski n]

"a Flow Registry that can hold extensions and we can then make nifi 2.0
fundamentally about distributing nifi as a kernel (small as possible) and
all extensions come from a flow registry on demand. "

Nice! Great to hear NiFi as an extensible platform. I would see that the
NiFi Registry is linked to a repository. Where one is the official
repository and another for community. At this repo/library people can
public put Processors/Services/Templates/Extensions. Browsing/Installing
should be similar to an app store (Jenkins does this nicely and also
Node-Red Library https://flows.nodered.org).

Another thing that I would like to see is that for developing/testing you
don't need to leave NiFi. For example directly code/debug
(codemirror/ace/monaco) and unit test it with input/output of a flowfile
for a specific processor.

Also a thing is making the GUI more responsive (rendering) and comfortable
(like undo). Most big features are already on Confluence:
https://cwiki.apache.org/confluence/display/NIFI/NiFi+Feature+Proposals


Raymond




Op vr 14 jun. 2019 om 22:59 schreef Peter Wicks (pwicks) :

> I've also heard something about a big change to the way relationships
> work. Maybe grouping relationships into larger groupings "failure" as a
> parent, with multiple optional/fine grained, children. Something like
> that.  If that rings a bell, maybe add it to your list .
>
> -Original Message-
> From: Joe Witt 
> Sent: Friday, June 14, 2019 11:31 AM
> To: dev@nifi.apache.org
> Subject: [EXT] Re: NiFi 2.0 Roadmap
>
> It makes sense there would be an mvp set of registry capability and thus a
> dependency for nifi 2.0 on registry readiness/version.  Otherwise I would
> largely hope not.
>
>
> On Fri, Jun 14, 2019 at 1:29 PM Otto Fowler 
> wrote:
>
> > Will that effort or planning be across all the nifi projects?  minifi
> > / cpp / registry etc?
> >
> >
> > On June 14, 2019 at 13:01:36, Joe Witt (joe.w...@gmail.com) wrote:
> >
> > Peter,
> >
> > Yeah I think we're all circling around similar thoughts on things
> > which are 'best for a major release' and we need to start codifying
> > that. At the same time we need this to be focused on items which can
> > only reasonably happen in a major release and not become a new kitchen
> > sink for JIRAs/ideas. We should frame up a wiki page for this effort.
> > I'm happy to kick that off soon (time permitting). In my mind the key
> > domino here is having a Flow Registry that can hold extensions and we
> > can then make nifi
> > 2.0 fundamentally about distributing nifi as a kernel (small as
> > possible) and all extensions come from a flow registry on demand.
> > Other obvious things like Java 11 as the base requirement and killing
> > off deprecated things come to mind.
> >
> > Thanks
> >
> > On Fri, Jun 14, 2019 at 11:45 AM Peter Wicks (pwicks)
> > 
> > wrote:
> >
> > > I've seen a lot of comments along the line of, "I don't think this
> > > will happen before NiFi 2.0". Do we have a roadmap/list somewhere of
> > > the big general changes planned for NiFi 2.0 or some kind of 2.0
> roadmap?
> > >
> > > --Peter
> > >
> >
>

Re: Best practices for working loosely-coupled and error handling

[ski n]

Thanks for the advice. Grouping multiple flows into one process group and
embedding process-groups within process groups are options to get some
oversight. They are limited however.

1) If multiple workflows are grouped into one Process Groups then
a) You need to create different output ports
or
b) connect multiple flows to the same output port

It's not allowed to have multiple output ports of the same name in one
Process Group.

2) Embedding has also limitations as the parent Process Group has no access
to the output port of the embedded Process Group.

Using multiple process groups is also not very practical. I have for
example a two phase processing of messages. First phase consists of about
ten process groups and second phase has also 10 process groups. Each
process group of the first phase can connect to all ten process groups of
the seconds phase. This leads to 100 relations crossing each other.

Basically the functionality is as I want, only it hard to find out what's
happening with so many process groups/ports connected. I see 2 options:

1) Use queues or topics on an external broker (JMS/AMQP/KAFKA)
2) Use Wormholes connections (When they get implemented...).
https://cwiki.apache.org/confluence/display/NIFI/Wormhole+Connections

I would prefer the second as then I don't need to use a third-party tool,
but for now will use the first one. In case someone has a better idea I
would like to hear of it of course.

Kind regards,

Raymond






Op wo 24 jul. 2019 om 21:42 schreef Mike Thomsen :

> More specifically, I mean wrap each of your 40 workflows in a process
> group. I have a workflow that processes some financial data, and it has 3
> levels of process groups at its most extreme points to group common
> functions and isolate edge cases so none of them are distracting when
> looking at the data flow from a higher level while it's running. It's about
> 100 processors total, but the canvas is quite clean because all of the
> functionality is neatly encapsulated in well-organized process groups that
> allow us to do things like add new sources and then drop them safely when
> they're no longer needed.
>
> On Wed, Jul 24, 2019 at 3:39 PM Mike Thomsen 
> wrote:
>
> > > In NiFi I can use ports, but than I need to connect those ports.
> >
> > You can wrap each operation in a process group and then connect the
> > process groups via ports so your main canvas is substantially less
> > cluttered. You can also nest process groups inside of each other. that
> > works really well for organizing related functionality.
> >
> > On Mon, Jul 22, 2019 at 10:17 AM ski n  wrote:
> >
> >> I work on migrating a large ESB process to a NiFi flow. This process
> >> contains around 40 events (40 different flowfiles). On the ESB a loosely
> >> coupled pattern was used with the help of JMS queues. In NiFi I can use
> >> ports, but than I need to connect those ports. The canvas soon becomes
> >> messy.
> >>
> >> Is there a way to use something like a ‘topic’ in Nifi? So some kind of
> >> endpoint without connecting items (processors/process groups) or is this
> >> against the dataflow concept and you always need external brokers like
> >> Kafka or ActiveMQ for this?
> >>
> >> Another question is what to do with failure messages. Can you configure
> a
> >> default ‘endpoint’ for all failures within a certain process.? Now I
> >> connect all processors to failure handling step/port, but this gets soon
> >> messy as well. What is the best practice for errors? Do most use
> >> autotermination?
> >>
> >>
> >> Regards,
> >>
> >>
> >> Raymond
> >>
> >
>

Re: Should there be a sane limit on Strings in NiFI?

[ski n]

Yes, it makes sense. But it shouldn't be too short as well. For example I
easily have RouteOnAttribute relationships with 1000+ characters.
(currently around 4000 is the longest).

Raymond

Op do 1 aug. 2019 om 17:05 schreef Joe Witt :

> Peter,
>
> It certainly makes sense to prevent authorized users from doing obviously
> unreasonable things in places where such filters could be applied.  This is
> a good example of such a thing.
>
> Thanks
>
> On Thu, Aug 1, 2019 at 10:52 AM Peter Wicks (pwicks) 
> wrote:
>
> > I noticed yesterday that automatic ellipsis was not working for
> > relationship names in the Connection Creation window (PR submitted in
> > NIFI-6512). As part of my test I started playing around with submitting
> > fairly long names for relationships in RouteOnAttribute.  This all worked
> > with out issue on relationships with greater than 1000 characters.
> >
> > This led me to wonder how other parts of NiFi deal with very long strings
> > in the UI, such as the breadcrumb trail.
> >
> > To bring an already long story to a close, I submitted a 200MB string as
> a
> > Process Group name using Chrome’s developer tools to do a custom PUT.
> NiFi
> > happily accepted the 104857600 character long name! Of course, this means
> > each call to load that process group fails spectacularly in Chrome with
> the
> > whole UI thread dead, but otherwise NiFi just keeps running.
> >
> > This is what led me to wonder if maybe we should have some constraints on
> > the maximum size of these UI visible strings.
> >
> > Thanks,
> >   Peter
> >
>

UI Features

[ski n]

I was creating a flow and I noticed some things of the GUI that were
missing. Don’t know if there already plans/tickets for these items, but I
would like your opinion first.

If you think it’s a good idea and there is no ticket yet than I will create
one:

1) It’s already possible to align flows, but sometimes it takes a lot
of time to make a flow look clean. I like if an automatically ordering
option of a flow (which calculate the best view). Ordering a flow from left
to right or ordering a flow from top to bottom.

2) Double-click on the canvas to zoom in.

3) Add possibility to use keyboard to move elements (for example
processors)

4) Select multiple items with mouse (like MS Visio). Now you need the
shift key to do this.

5) Moving step/flows into a process group. “Right-Click” on a flow step
(Move step to process group / Move flow to process group / Create process
group for this flow)

6) Possibility to add links in labels (for example link to a process
group)

7) Search box (to find and navigate to a process group/flow/flow step)

8) Queues are highlighted yellow, while processors are highlight black
(Better both yellow, black is hard to see)

9) Add Ctrl X / Cut

10)  Add Ctrl Z / Revert last change

11)  Sometimes the "apply" button navigates you directly back to the canvas
(for example with configuration), sometimes not (process group). Like to
have a 'save' button which means apply change and leave the popup, while
apply applies the change.

12)  Export canvas to image (jpeg/png)

13)  Links to file resources --> open the resource in an online editor

14) Option to edit the processor as XML (sometimes this goes faster). Now
you need to export/import the processor as a template.



Best regards,


Raymond

Best practices for working loosely-coupled and error handling

[ski n]

I work on migrating a large ESB process to a NiFi flow. This process
contains around 40 events (40 different flowfiles). On the ESB a loosely
coupled pattern was used with the help of JMS queues. In NiFi I can use
ports, but than I need to connect those ports. The canvas soon becomes
messy.

Is there a way to use something like a ‘topic’ in Nifi? So some kind of
endpoint without connecting items (processors/process groups) or is this
against the dataflow concept and you always need external brokers like
Kafka or ActiveMQ for this?

Another question is what to do with failure messages. Can you configure a
default ‘endpoint’ for all failures within a certain process.? Now I
connect all processors to failure handling step/port, but this gets soon
messy as well. What is the best practice for errors? Do most use
autotermination?


Regards,


Raymond

Processor convert between data formats / Community repository

[ski n]

Hi all,

I made a DocConverter to convert between different data formats
(XML,JSON,CSV,YAML). For those interested there is also a NiFi Processor:

https://github.com/assimbly/convertdataformat
https://github.com/assimbly/docconverter

I also wondered if there is a repository or list where to place
cummunity/custom NiFi processors. Where you provide things like:

- Documentation
- Link to code
- Nar file

I only now about https://github.com/jfrazee/awesome-nifi where a lot of
custom processor are placed. But that is not actively maintained.

Would it be an idea to add a community repository where one could download
custom processors (or more exotic processor not part of default
distribution/core)
directly for the NiFI GUI (similar to for example Jenkins plugins)?

Raymond

Re: [DISCUSS] Early, voluntary relocation to GitBox

[ski n]

Maybe good to check the link on the NiFi website (Development --> Source).
It still points out to "https://git-wip-us.apache.org/repos/asf?p=nifi.git;

Raymond

Op vr 14 dec. 2018 om 15:13 schreef Aldrin Piri :

> And in my frenzied typing, I made use of nifi-site's locations instead of
> NiFi.
>
> NiFi is available at Gitbox for checkout at
> https://gitbox.apache.org/repos/asf/nifi.git with its web view at
> https://gitbox.apache.org/repos/asf?p=nifi.git
>
> On Fri, Dec 14, 2018 at 8:23 AM Aldrin Piri  wrote:
>
> > Some clarifying notes until I can get all the updates into place and
> > generate a new email.
> >
> > There are now two locations that are writable by committers for each of
> > our repositories.  The updated repository locations can be seen in the
> > Apache NiFi section of https://gitbox.apache.org/repos/asf.  NiFi, for
> > instance, is available at
> > https://gitbox.apache.org/repos/asf/nifi-site.git.  The web view of the
> > repository is located at
> > https://gitbox.apache.org/repos/asf?p=nifi-site.git.  Please note the
> > subtle distinction!
> >
> > If you are a contributor still on their way to committership or you are
> > committer content with the same functionality we had with our old setup
> > (git-wip), you will use the above listed locations.  You can update your
> > location by performing the following steps:
> >
> >- Go to the root of your source directory, we'll use nifi as an
> example
> >   - cd ${repo_home}/nifi
> >- Update your remote.  On a default checkout this will be origin, but
> >use the name that is pointing to the old git-wip location (you can
> view
> >these using git remote -v)
> >   - git remote set-url origin
> >   https://gitbox.apache.org/repos/asf/nifi.git
> >
> >
> > If you are a committer and interested in making use of the tighter GitHub
> > integration, there are a few steps needed to enable this described at
> > https://gitbox.apache.org/setup/.  Please follow those and give some
> time
> > to allow the background processes sync.  Successfully completing them
> > should result in the email from GitHub on behalf of Gitbox I mentioned
> > previously.
> >
> > Again, look for a bit more polished set of instructions in the next hour
> > or so but feel free to add any questions here in the interim if it is
> > holding you up.
> >
> >
> >
> > On Fri, Dec 14, 2018 at 7:35 AM Aldrin Piri 
> wrote:
> >
> >> Hey all,
> >>
> >> Just a quick note that migration has completed.  I'll be gathering up
> >> information shortly and sending out another, separate email to the
> >> community with any needed changes and will work to update our docs
> >> appropriately.
> >>
> >> Committers, you should have received an email regarding you addition to
> >> the nifi group on GitHub.
> >>
> >> --aldrin
> >>
> >> On Thu, Dec 13, 2018 at 1:20 PM Aldrin Piri 
> wrote:
> >>
> >>> Hey folks,
> >>>
> >>> The JIRA ticket has been submitted [1].  I will keep this thread
> updated
> >>> as things progress and then generate a separate "helper" email for
> >>> instructions/any updates that may be needed upon completion.
> >>>
> >>> [1] https://issues.apache.org/jira/browse/INFRA-17419
> >>>
> >>> On Mon, Dec 10, 2018 at 10:51 AM Pierre Villard <
> >>> pierre.villard...@gmail.com> wrote:
> >>>
>  +1 as well, should ease the contribution workflow.
>  Thanks for volunteering Aldrin.
> 
>  Le lun. 10 déc. 2018 à 16:41, Laszlo Horvath <
> lhorv...@hortonworks.com>
>  a
>  écrit :
> 
>  > +1
>  >
>  > On 10/12/2018, 15:22, "Michael Moser"  wrote:
>  >
>  > +1 from me, sounds great.
>  >
>  >
>  > On Mon, Dec 10, 2018 at 4:07 AM Arpad Boda <
> ab...@hortonworks.com
>  >
>  > wrote:
>  >
>  > > +1 (for being the guinea pig __ )
>  > >
>  > > On 09/12/2018, 04:01, "Aldrin Piri" 
>  wrote:
>  > >
>  > > Thanks to those of you that responded.
>  > >
>  > > I think my tentative plan is to give this a few more days
>  to see
>  > if
>  > > there
>  > > are any strong objections.  Otherwise, I'll look to file
> the
>  > ticket
>  > > around
>  > > midweek and then look to begin the process toward the end
> of
>  > next week
>  > > to
>  > > hopefully minimize disruptions.  Thanks again!
>  > >
>  > > On Sat, Dec 8, 2018 at 12:09 PM Tony Kurc <
> trk...@gmail.com
>  >
>  > wrote:
>  > >
>  > > > +1, sounds like a great idea
>  > > >
>  > > > On Sat, Dec 8, 2018 at 7:44 AM Mike Thomsen <
>  > mikerthom...@gmail.com>
>  > > > wrote:
>  > > >
>  > > > > +1
>  > > > >
>  > > > > On Fri, Dec 7, 2018 at 9:08 PM Sivaprasanna <
>  > > sivaprasanna...@gmail.com>
>  > > > > wrote:
>  > > > >
>