Re: NiFi 2.0 - QuestDB

[Brandon DeVries]

I also agree... I've been doing some testing recently, and not having to note 
relevant stats before a config change and restart is a huge improvement.

From: Pierre Villard 
Sent: Wednesday, July 19, 2023 8:54:13 AM
To: dev@nifi.apache.org 
Subject: Re: NiFi 2.0 - QuestDB

I do think this provides great value. The possibility to get access to
status history of the components and at system level across restart is a
great improvement for NiFi troubleshooting. It also gives the ability to
store this information for a longer period of time. I'm definitely in favor
of making this the default starting with NiFi 2.0.

Le mer. 19 juil. 2023 à 13:49, Simon Bence  a
écrit :

> Hi Community,
>
> I was thinking if it would make sense to set the QuestDB as default for
> status history backend in 2.0? It is there for a while and I would consider
> it as a step forward so the new major version might be a good time for the
> wider audience. It comes with less memory usage for bigger flows, the
> possibility of checking status information when the node is not running or
> restarted so I think it worth consideration. Any insight or improvement
> point is appreciated, thanks!
>
> Regards,
> Bence

Re: [discuss] nifi 2.0 and Java 21…

[Brandon DeVries]

+1 to requiring java 21. Starting off as "up to date" as possibly makes a lot 
of sense, and some of the new features seem especially relevant to NiFi.

I definitely understand the concerns about organizations being willing / able 
to approve Java 21... But those same organizations might also be hesitant to 
move to NiFi 2.0. We will continue to support java 17 & NiFi 1.x for some time, 
so hopefully those groups will have the time they need to get approvals, do 
evaluations, and upgrade.

Brandon

From: Pierre Villard 
Sent: Thursday, September 7, 2023 6:15:58 AM
To: dev@nifi.apache.org 
Subject: Re: [discuss] nifi 2.0 and Java 21…

Hi all,

I share the concerns raised by Chris regarding how quickly users of NiFi
will be able to adopt Java 21.

While I'm definitely in favor of using the latest and greatest, especially
when it brings to the table such significant features, we need to be
careful as it may significantly delay the adoption of NiFi 2.0 in big
companies where deploying Java 21 will take time. I acknowledge that going
from Java 8 to Java 17 is certainly the same effort as going from Java 8 to
Java 21 but how quickly security-sensitive environments will adopt a new
release of Java that is completely new?

In addition to that, it sounds like we would add a significant rework of
the framework in NiFi 2.0 assuming we adopt Java 21 as the minimum version.
Do we think this is going to significantly delay the first release of NiFi
2.0? We still have work to do but adding this on top could delay the
release, no?

Finally, the features that Java 21 are bringing sound super interesting in
the context of NiFi but do we already have an idea of what it will improve?
is it the user experience, and if so, how will it change? is it the
performance, and if so, do we have an idea of how things will improve?

Thanks,
Pierre

Le mer. 6 sept. 2023 à 23:07, Chris Sampson
 a écrit :

> Yeah, I understand the need to move to 21 as a minimum to take advantage of
> its features. Hopefully the wider java ecosystem won't be an issue in the
> short term.
>
> I just wanted the discussion to be clear about this being a change to the
> Java baseline/minimum for NiFi 2.0.
>
> It's a +1 from me.
>
> On Wed, 6 Sept 2023, 19:01 Joe Witt,  wrote:
>
> > Chris
> >
> > My suggestion is rooted in making Java 21 the minimum of the NiFi 2.0
> > line.  It would not work on Java 17.  The reason for this is so that we
> can
> > leverage the longest duration of a given LTS line while also benefiting
> > from the language improvements that affords.  Maintaining compatibility
> > with future versions we generally have to do.  But whatever the minimum
> > version we accept dictates which language features we can leverage.  So
> if
> > it is 17 then we can't leverage anything from the 21 line for example.
> >
> > GIven the nature and timelines of LTS I don't really think there is the
> > same burn in logic that we'd have all known in the past before the
> > LTS/STS/etc.. release constructs existed.
> >
> > Thanks
> >
> > On Wed, Sep 6, 2023 at 10:53 AM Chris Sampson
> >  wrote:
> >
> > > To be clear, is the discussion one of making Java 21 the minimum
> > > requirement for NiFi 2.0.0, or rather NiFi 2.x be compatible with Java
> > 21,
> > > while retaining Java 17 as a minimum?
> > >
> > > If we moved straight to a Java 21 requirement, will we run into
> > > compatibility issues that delay initial NiFi 2 release? Will the move
> to
> > > Java 21 mean some organisations delay their migration to NiFi 2 through
> > not
> > > wanting to move to the latest Java LTS version until it's had a time
> for
> > > "settling" through security/bug patches, etc.? And are either of these
> > > sufficient concern to pause Java 21 becoming the requirement, as we may
> > > then need to extend NiFi 1.x maintenance for longer into the future?
> > >
> > > Generally, I'm in favour of moving to "latest and greatest",
> particularly
> > > for LTS versions of technologies, but the rate of Java version adoption
> > > across the community gives me pause.
> > >
> > > I certainly see the advantage of new Java features for NiFi in Java 21,
> > > such as the already mentioned virtual threads.
> > >
> > > On Wed, 6 Sept 2023, 18:34 Mike Thomsen, 
> wrote:
> > >
> > > > +1 100%
> > > >
> > > > On Wed, Sep 6, 2023 at 11:48 AM Adam Taft  wrote:
> > > >
> > > > > Yes, please. +1 Exactly what Mark said. Virtual threads have
> > potential
> > > to
> > > > > be extremely impactful to applications like NiFi.
> > > > >
> > > > > /Adam
> > > > >
> > > > > On Wed, Sep 6, 2023 at 7:26 AM Mark Payne 
> > > wrote:
> > > > >
> > > > > > Thanks for bringing his up, Joe.
> > > > > >
> > > > > > I would definitely be a +1. I think the new virtual thread
> concept
> > > > would
> > > > > > have great impact on us.
> > > > > > It would allow us to significantly simplify our scheduling logic,
> > > which
> > > > > > would help with code maintainability
> > > > > > but

Re: new PackageFlowFile processor

[Brandon DeVries]

I have had to use that pattern myself recently. I think a simple 
PackageFlowFile processor makes a lot of sense. I am +1.

Brandon

From: Michael Moser 
Sent: Friday, September 8, 2023 9:52:52 AM
To: dev@nifi.apache.org 
Subject: new PackageFlowFile processor

Devs,

I can't find if this was suggested before, so here goes.  With the demise
of PostHTTP in NiFi 2.0, the recommended alternative is to MergeContent 1
file into FlowFile-v3 format then InvokeHTTP.  What does the community
think about supporting a new PackageFlowFile processor that is simple to
configure (compared to MergeContent!) and simply packages flowfile
attributes + content into a FlowFile-v[1,2,3] format?  This would also
offer a simple way to export flowfiles from NiFi that could later be
re-ingested and recovered using UnpackContent.  I don't want to submit a PR
for such a processor without first asking the community whether this would
be acceptable.

Thanks,
-- Mike

Re: new PackageFlowFile processor

[Brandon DeVries]

Most of the complexity in MergeContent is around the bundling parameters...
this processor would do no bundling, just straight pass through to the
packaging library.  No worries for the user about setting max package size,
number of entries, number of bins, bin age, headers, footers, etc... even
if they have sensible defaults in merge content, it's potentially confusing
that if I want to "package" a FlowFile, what I need to do is "merge" it
with itself (and ignore all the other confusing / irrelevant settings).
Bypassing the bundling logic probably improves performance for this use
case as well.

On Fri, Sep 8, 2023 at 9:56 AM Joe Witt  wrote:

> Mike
>
> In user terms this makes sense to me. Id only bother with v3 or whatever is
> latest. We want to dump the old code. And if there are seriously older
> versions v1,v2 then nifi 1.x can be used.
>
> The challenge is that you end up needing some of the same complexity in
> implementation and config of merge content i think. What did you have in
> mind for that?
>
> Thanks
>
> On Fri, Sep 8, 2023 at 6:53 AM Michael Moser  wrote:
>
> > Devs,
> >
> > I can't find if this was suggested before, so here goes.  With the demise
> > of PostHTTP in NiFi 2.0, the recommended alternative is to MergeContent 1
> > file into FlowFile-v3 format then InvokeHTTP.  What does the community
> > think about supporting a new PackageFlowFile processor that is simple to
> > configure (compared to MergeContent!) and simply packages flowfile
> > attributes + content into a FlowFile-v[1,2,3] format?  This would also
> > offer a simple way to export flowfiles from NiFi that could later be
> > re-ingested and recovered using UnpackContent.  I don't want to submit a
> PR
> > for such a processor without first asking the community whether this
> would
> > be acceptable.
> >
> > Thanks,
> > -- Mike
> >
>

Re: Is there a configuration to limit the size of nifi's flowfile repository

[Brandon DeVries]

All,

This is something I think we shouldn't dismiss so easily.  While the
FlowFile repo is lighter than the content repo, allowing it to grow too
large can cause major problems.

Specifically, an "overgrown" FlowFile repo may prevent a NiFi instance from
coming back up after a restart due to the way in which records are held in
memory.  If there is more memory available to give to the JVM, this can
sometimes be worked around... but if there isn't you may just be out of
luck.  For that matter, allowing the FlowFile repo to grow so large that it
consumes all the heap isn't going to be good for system health in general
(OOM is probably never where you want to be...).

To Pierre's point "you don't want to limit that repository in size since it
would prevent the workflows to create new flow files"... that's exactly why
I would want to limit the size of the repo.  You do then get into questions
of how exactly to do this.  For example, you may not want to simply block
all transactions that create a FlowFile, because it may remove even more
(e.g. MergeContent).  Additionally, you have to be concerned about
deadlocks (e.g. a "Wait" that hangs forever because its "Notify" is being
starved).  Or, perhaps that's all you can do... freeze everything at some
threshold prior to actual damage being done, and alert operators that
manual intervention is necessary (e.g. bring up the graph with
autoResume=false, and bleed off data in a controlled fashion).

In summary, I believe this is a problem.  Even if it doesn't come up often,
when it does it is significant.  While the solution likely isn't simple,
it's worth putting some thought towards.

Brandon

On Wed, Apr 25, 2018 at 9:43 AM Sivaprasanna 
wrote:

> No, he actually had mentioned “like content repository”. The answer is,
> there aren’t any properties that support this, AFAIK. Pierre’s response
> pretty much sums up why there aren’t any properties.
>
> Thanks,
> Sivaprasanna
>
> On Wed, 25 Apr 2018 at 7:10 PM, Mike Thomsen 
> wrote:
>
> > I have a feeling that what Ben meant was how to limit the content
> > repository size.
> >
> > On Wed, Apr 25, 2018 at 8:26 AM Pierre Villard <
> > pierre.villard...@gmail.com>
> > wrote:
> >
> > > Hi Ben,
> > >
> > > Since the flow file repository contains the information of the flow
> files
> > > currently being processed by NiFi, you don't want to limit that
> > repository
> > > in size since it would prevent the workflows to create new flow files.
> > >
> > > Besides this repository is very lightweight, I'm not sure it'd need to
> be
> > > limited in size.
> > > Do you have a specific use case in mind?
> > >
> > > Pierre
> > >
> > >
> > > 2018-04-25 9:15 GMT+02:00 尹文才 :
> > >
> > > > Hi guys, I checked NIFI's system administrator guide trying to find a
> > > > configuration item so that the size of the flowfile repository could
> be
> > > > limited similar to the other repositories(e.g. content repository),
> > but I
> > > > didn't find such configuration items, is there currently any
> > > configuration
> > > > to limit the flowfile repository's size? thanks.
> > > >
> > > > Regards,
> > > > Ben
> > > >
> > >
> >
>

Re: JSON Path Expression

[Brandon DeVries]

I happened to be playing with this Friday... You can try single quotes in
the jsonPath as well.

Brandon
On Fri, Apr 27, 2018 at 5:05 PM Otto Fowler  wrote:

> $.fields[?(@.name==Type)].maskType
>
>
> no \” \”
>
> When JsonPath evaluates the path against the content, the content has
> already been
> parsed into a Map.
> So, when Jackson parses the json   “name” : “Type”  it ends up in a map
> that has those strings,
> but NOT the literal quotes in the string.
>
> When you pass in \”Type\”  it ends up doing
> if ( “Type” == “\”Type\”” )  ** not with the \ but the embedded quotes
>
> and  that is false.
>
>
> On April 27, 2018 at 16:41:51, Anil Rai (anilrain...@gmail.com) wrote:
>
> Thanks Otto.
> I did try changing the json path to :
> $.fields[?(@.name==\"Type\")].maskType
> I am still getting [] output.
> I am I missing something?
>
>
> On Fri, Apr 27, 2018 at 4:32 PM, Otto Fowler 
> wrote:
>
> > For the record, I am not saying that the json path library version nifi
> > uses isn’t broken because this doesn’t work.
> >
> >
> >
> > On April 27, 2018 at 16:30:57, Otto Fowler (ottobackwa...@gmail.com)
> > wrote:
> >
> > The reason why your path is failing is because of the embedded quotes.
> > If you remove the quotes it will work.
> > I wrote this test:
> >
> >
> > @Test
> > public void testIssue() throws Exception{
> > String jsonPathAttrKey = "JsonPath1";
> > final TestRunner testRunner = TestRunners.newTestRunner(new
> > EvaluateJsonPath());
> > testRunner.setProperty(EvaluateJsonPath.DESTINATION,
> > EvaluateJsonPath.DESTINATION_CONTENT);
> > testRunner.setProperty("JsonPath1", "$.fields[?(@.name==Type)].
> > maskType");
> >
> > testRunner.enqueue(JSON_FAIL_SNIPPET);
> > testRunner.run();
> >
> > Relationship expectedRel = EvaluateJsonPath.REL_MATCH;
> >
> > testRunner.assertAllFlowFilesTransferred(expectedRel, 1);
> > final MockFlowFile out =
> > testRunner.getFlowFilesForRelationship(expectedRel).get(0);
> > out.assertContentEquals("[\"b\"]");
> > }
> >
> >
> > Here, JSON_FAIL_SNIPPET is your content as a file.
> >
> > I understand that your original path works on the test web sites. It
> > *also* works if you call JsonPath differently:
> >
> > /**{ "fields":[
> > {
> > "maskType": "a",
> > "name": "Id"
> > },
> > {
> > "maskType": "b",
> > "name": "Type"
> > }
> > ]
> > }*/
> > @Multiline
> > static String NIFI_PROB;
> >
> > @Test
> > public void testNifi() {
> > App app = new App();
> > List value = app.justJsonPath(NIFI_PROB.getBytes(),
> > "$.fields[?(@.name==\"Type\")].maskType");
> > if( value != null ) {
> > value.forEach(new Consumer() {
> > @Override
> > public void accept(Object o) {
> > System.out.println(o.toString());
> > }
> > });
> > }
> > }
> >
> > private final ObjectMapper objectMapper = new ObjectMapper();
> > public App() {
> > Configuration.setDefaults(new Configuration.Defaults() {
> >
> > private final JsonProvider jsonProvider = new JacksonJsonProvider();
> > private final MappingProvider mappingProvider = new
> > JacksonMappingProvider();
> >
> > @Override
> > public JsonProvider jsonProvider() {
> > return jsonProvider;
> > }
> >
> > @Override
> > public MappingProvider mappingProvider() {
> > return mappingProvider;
> > }
> >
> > @Override
> > public Set options() {
> > return EnumSet.noneOf(Option.class);
> > }
> > });
> > CacheProvider.setCache(new LRUCache(100));
> > }
> >
> > public List justJsonPath(byte[] rawMessage, String jsonPath) {
> > return JsonPath.parse(new String(rawMessage))
> > .read(jsonPath);
> > }
> >
> > b
> >
> > Process finished with exit code 0
> >
> > But the way the Nifi uses JsonPath you need to call it as such.
> >
> > Hope this helps.
> >
> > ottO
> >
> >
> > On April 27, 2018 at 16:09:32, Mike Thomsen (mikerthom...@gmail.com)
> > wrote:
> >
> > The jayway JSONPath library on GitHub is the library I was referring to.
> > I'll check on Jira to see if there's a ticket. Seems like something we
> > could do for 1.7
> >
> > Thanks,
> >
> > Mike
> >
> > On Fri, Apr 27, 2018 at 3:46 PM Anil Rai  wrote:
> >
> > > Hi Mike,
> > > Thanks for your quick reply. I am using nifi 1.5.0 release. I am not
> sure
> > > what you mean by "trying against the latest version of java lib?"
> > > If you could further elaborate please?
> > > Also is there any other way to achieve this outcome (either by changing
> > the
> > > path or by using a completely different processor other than
> > > evaluatejsonpath?)
> > >
> > > Thanks
> > > Anil
> > >
> > >
> > > On Fri, Apr 27, 2018 at 3:37 PM, Mike Thomsen 
> > > wrote:
> > >
> > > > The JsonPath processor uses an older version of the Java library. I
> > think
> > > > it's v2.0. Have you tried that against the latest version of the Java
> > > lib?
> > > > I know there's at least one JsonPath feature the version we use
> doesn't
> > > > support.
> > > >
> > > > On Fri, Apr 27, 2018 at 3:14 PM Anil Rai 
> > wrote:
> > > >
> > > > > Experts,
> > > > >
> > > > > Input JSON
> > > > > ---
> > > > > { "fields":[
> > > > > {
> > >

Re: [EXT] Re: Primary Only Content Migration

[Brandon DeVries]

In this particular case, the easiest thing to do may be to keep track of
the hostname of the original node of the file (before split and
distribution).  Then, just send all of the pieces back to the originator
node when done, and merge there.  It would be nice to have automatic queue
balancing of some sort, but that's going to be a major effort, and not
really necessary for Peter's issue.

Brandon

On Fri, Jun 8, 2018 at 10:25 AM Bryan Bende  wrote:

> I'm also curious to know how Peter is currently handling this since he
> mentioned already sending back to primary node, and we don't yet have
> any of the functionality that Joe, Koji, and Pierre mentioned, so the
> only way I can think of doing that would be to use PostHttp/InvokeHttp
> and point directly at the URL of the primary node, although that would
> be problematic when it changes.
>
> For Peter's scenario it sounds like he wants to run MergeContent in
> defragment mode on the primary node after converging all the splits to
> that node. In that case it may be challenging to handle the scenarios
> like Koji mentioned... if we are half way through converging
> everything to primary node and then primary node changes, do we
> automatically move all of the data that was already sent to the old
> primary over to the new primary so the defragment can continue
> successfully? Maybe there is a special type of "merge" or "reduce"
> component that can work across the cluster and handle this.
>
>
> On Fri, Jun 8, 2018 at 4:01 AM, Pierre Villard
>  wrote:
> > Hi guys,
> >
> > Koji is right, I initially filed NIFI-4026 to cover this kind of use
> case.
> >
> > There is a lot of challenges and ways to address this subject.
> > Auto-balancing in queues will be a super nice way forward this goal.
> >
> > I think an easy first step would be to add a checkbox in RPG
> configuration
> > allowing the user to specifically send the data to the remote primary
> node
> > only. That's an easy information to add in S2S peer status data requested
> > by clients. It would need proper documentation when we have nodes
> > disconnecting/reconnecting, but I think that's the "easiest" improvement
> to
> > achieve your goal here.
> >
> > Since the changes are rather complex, I think we need to carefully think
> > about the solution design here.
> >
> > Pierre
> >
> >
> >
> > 2018-06-08 3:48 GMT+02:00 Koji Kawamura :
> >
> >> There is an existing JIRA submitted by Pierre.
> >> I think its goal is the same with what Joe mentioned above.
> >> https://issues.apache.org/jira/browse/NIFI-4026
> >>
> >> As for hashing and routing data with affinity/correlation, I think
> >> 'Consistent Hashing' is the most popular approach to minimize the
> >> impact of node addition/deletion.
> >> Applying Consistent Hashing to S2S client may not be difficult. The
> >> challenging part is how to support cluster topology change in the
> >> middle of transferring data that needs correlation.
> >>
> >> A simple challenging scenario:
> >> Let's say there is a group of 4 FlowFiles having correlation id as
> 'rel-A'
> >> 1. Client sends rel-A, data-1of4 to Node1
> >> 2. Client sends rel-A, data-2of4 to Node1
> >> 3. NodeN is added and it takes some part in hash key space that Node1
> >> was assigned to
> >> 4. Client sends rel-A, data-3of4 to NodeN
> >> 5. Client sends rel-A, data-4of4 to NodeN
> >>
> >> Then, a Merge processor running on Node1 and NodeN can not complete
> >> because it won't have the whole dataset to merge.
> >> This situation can be handled manually if we document it well.
> >> Or adding resending loop, so that:
> >>
> >> 5. Client on Node1 resends rel-A, data1of4 to NodeN
> >> 6. Client on Node1 resends rel-A, data2of4 to NodeN
> >> 7. Merge processor on NodeN merges the FlowFiles.
> >>
> >> I'm interested in working on this improvement, too.
> >>
> >> Thanks,
> >> Koji
> >>
> >>
> >> On Fri, Jun 8, 2018 at 8:19 AM, Joe Witt  wrote:
> >> > Peter
> >> >
> >> > I'm not sure there is a good way for a processor to drive such a thing
> >> > with existing infrastructure.  The processor having ability to know
> >> > about the structure of a cluster is not something we have wanted to
> >> > expose for good reasons.  There would likely need to be a more
> >> > fundamental point of support for this.
> >> >
> >> > I'm not sure what that design would look like just yet - but agreeing
> >> > this is an important step to take soon.  If you want to start
> >> > sketching out design ideas that would be awesome.
> >> >
> >> > Thanks
> >> > On Thu, Jun 7, 2018 at 6:11 PM Peter Wicks (pwicks) <
> pwi...@micron.com>
> >> wrote:
> >> >>
> >> >> Joe,
> >> >>
> >> >> I agree it is a lot of work, which is why I was thinking of starting
> >> with a processor that could do some of these operations before looking
> >> further. If the processor could move flowfile's between nodes in the
> >> cluster it would be a good step. Data comes in form a queue on any node,
> >> but gets written out to a queue on only the desire

Re: NiFi (UNCLASSIFIED)

[Brandon DeVries]

Justin,

The 4.x version you're referencing is specific to the organization you work
for.  I would direct questions specific to this version to your
organization's internal chatrooms / DLs.  If you need further guidance,
please let me know.

Brandon

On Tue, Jun 26, 2018 at 3:49 PM Cetron, Justin F CTR USARMY CECOM (US) <
justin.f.cetron@mail.mil> wrote:

> CLASSIFICATION: UNCLASSIFIED
>
> Good Afternoon,
>
> I had a question in regards to NiFi. I recently have been tasked with
> upgrading the NiFi we run, however, in looking at the versions I have vs.
> what I have seen on the nifi.apache.org site, are there two different
> versions of the software? As what I was handed is 3 full generations higher
> (4.x) than the 1.7 I see on the normal site. Thank you very much.
>
>
> Justin Cetron - Systems Administrator III
> Office: (443) 861-4215
>
> CLASSIFICATION: UNCLASSIFIED
>

Re: [discuss] should we do a nifi 1.7.1 release?

[Brandon DeVries]

This one may be worth fixing in 1.7.1 as well:

https://issues.apache.org/jira/browse/NIFI-5331



On Thu, Jul 5, 2018 at 12:23 PM Joe Witt  wrote:

> team,
>
> Wanted to kick off a thread to suggest we do a nifi 1.7.1 release.  It
> sounds like we might have an issue handling wildcard certs in 1.7.0
> [1] and it was reported again in an email today i think.  Also, if
> this one is deemed legit it seems worth sorting out [2].  I'd imagine
> there are a few other bug fixes as well we can pull in.
>
>
> [1] https://issues.apache.org/jira/browse/NIFI-5370
> [2] https://issues.apache.org/jira/browse/NIFI-5377
>
> Thanks
> Joe
>

Re: Dark mode

[Brandon DeVries]

I think there are a lot of people that would be a big +1 on this.  Maybe
even more so if it was abstracted so there could be multiple / custom
"themes" (e.g. dark, classic, high contrast / 508 compliant...).

Brandon

On Fri, Jul 13, 2018 at 7:29 AM Joe Witt  wrote:

> Rich
>
> This could certainly be an interesting option.  Perhaps file a JIRA
> with some details on where you're at in your PR and post that when
> ready.  Identify things you think might be remaining and such..
>
> Thanks
>
> On Fri, Jul 13, 2018 at 7:25 AM, Rich M  wrote:
> > Hi
> >
> > Going through some NiFi bits and pieces while I'm between projects;
> > I've got a half-baked dark mode kicking around, is there any interest
> > in me pushing this to a remote branch somewhere? There's some iframes
> > in dialogs missing styling, it probably wants someone more familiar
> > with Angular to look at it and a couple of places the styling isn't
> > applied to so it'd need a little work to even consider merging.
> >
> > https://i.imgur.com/lSofSq5.jpg
> >
> > Rich
>

Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases

[Brandon DeVries]

In regards to "We 'could' also split out the 'nifi-api'...", NiFi 2.0 would
also be a good time to look at more clearly defining the separation between
the UI and the framework.  Where nifi-api is the contract between the
extensions and the framework, the NiFi Rest api is the contract between the
UI and framework...  These pieces could potentially be built  / deployed /
updated independently.

On Thu, May 30, 2019 at 11:39 AM Jeff  wrote:

> In the same category of challenges that Peter pointed out, it might be
> difficult for Travis to build the "framework" and "extensions" projects if
> there are changes in a PR that affect both projects.
>
> Is there a good way in Travis to have the workspace/maven repo shared
> between projects in a single build?
>
> It's probably always in the direction of the extensions project needing
> something new to be added to the framework project rather than the other
> way around, but it'll be tricky to get that working right in Travis if it's
> not possible to set up the Travis build to know it needs to deploy the
> framework project artifacts into a maven repo that the extension project
> will use.
>
> One way might be to make sure that changes to the framework project must be
> in master before the extensions project can make use of them, but that
> would require a "default master" build for the framework project which
> builds master after each commit, and deploys the build artifacts to a
> persistent maven repo that the extension project builds can access.  It
> also makes project-spanning change-sets take longer to review and get fully
> committed to master.
>
> On Thu, May 30, 2019 at 11:23 AM Peter Wicks (pwicks) 
> wrote:
>
> > One more "not awesome" would be that core changes that affect extensions
> > will be a little harder to test. If I make a core change that changes the
> > signature of an interface/etc... I'll need to do some extra work to make
> > sure I don't break extensions that use it.
> >
> > Still worth it, just one more thing to mention.
> >
> > -Original Message-
> > From: Joe Witt 
> > Sent: Thursday, May 30, 2019 9:19 AM
> > To: dev@nifi.apache.org
> > Subject: [EXT] [discuss] Splitting NiFi framework and extension repos and
> > releases
> >
> > Team,
> >
> > We've discussed this a bit over the years in various forms but it again
> > seems time to progress this topic and enough has changed I think to
> warrant
> > it.
> >
> > Tensions:
> > 1) Our build times take too long.  In travis-ci for instance it takes 40
> > minutes when it works.
> > 2) The number of builds we do has increased.  We do us/jp/fr builds on
> > open and oracle JDKs.  That is 6 builds.
> > 3) We want to add Java 11 support such that one could build with 8 or 11
> > and the above still apply.  The becomes 6 builds.
> > 4) With the progress in NiFi registry we can now load artifacts there and
> > could pull them into NiFi.  And this integration will only get better.
> > 5) The NiFi build is too huge and cannot grow any longer or else we
> cannot
> > upload convenience binaries.
> >
> > We cannot solve all the things just yet but we can make progress.  I
> > suggest we split apart the NiFi 'framework/application' in its own
> release
> > cycle and code repository from the 'nifi extensions' into its own
> > repository and release cycle.  The NiFi release would still pull in a
> > specific set of extension bundles so to our end users at this time there
> is
> > no change. In the future we could also just stop including the extensions
> > in nifi the application and they could be sourced at runtime as needed
> from
> > the registry (call that a NiFi 2.x thing).
> >
> > Why does this help?
> > - Builds would only take as long as just extensions take or just core/app
> > takes.  This reduces time for each change cycle and reduces load on
> > travis-ci which runs the same tests over and over and over for each pull
> > request/push regardless of whether it was an extension or core.
> >
> > - It moves us toward the direction we're heading anyway whereby
> extensions
> > can have their own lifecycle from the framework/app itself.
> >
> > How is this not awesome:
> > - Doesn't yet solve for the large builds problem.  I think we'll get
> there
> > with a NiFi 2.x release which fully leverages nifi-registry for retrieval
> > of all extensions.
> > - Adds another 'thing we need to do a release cycle for'.  This is
> > generally unpleasant but it is paid for once a release cycle and it does
> > allow us to release independently for new cool extensions/fixes apart
> from
> > the framework itself.
> >
> > Would be great to hear others thoughts if they too feel it is time to
> make
> > this happen.
> >
> > Thanks
> > Joe
> >
>

Re: [EXT] Re: [VOTE] Create NiFi Standard Libraries sub-project

[Brandon DeVries]

+1, binding

Brandon

From: Mark Payne 
Sent: Wednesday, September 4, 2019 1:35 PM
To: dev@nifi.apache.org
Subject: Re: [EXT] Re: [VOTE] Create NiFi Standard Libraries sub-project

I'm a +1 as well.

Thanks
-Mark


> On Sep 4, 2019, at 6:02 AM, Pierre Villard  
> wrote:
>
> +1 (binding)
>
> As a minor comment - but definitely not the place to discuss it - I wonder
> if having a dedicated JIRA for that is really required. We could have the
> same approach as we do with nifi-fds (use the NIFI JIRA with the right use
> of tags, components, versions).
>
> Le mer. 4 sept. 2019 à 08:22, Jeff  a écrit :
>
>> +1 Create NiFi Standard Libraries (binding)
>>
>> On Wed, Sep 4, 2019 at 12:19 AM Peter Wicks (pwicks) 
>> wrote:
>>
>>> +1, binding
>>>
>>> -Original Message-
>>> From: Kevin Doran 
>>> Sent: Tuesday, September 3, 2019 7:12 PM
>>> To: dev@nifi.apache.org; dev@nifi.apache.org
>>> Subject: [EXT] Re: [VOTE] Create NiFi Standard Libraries sub-project
>>>
>>> +1, binding
>>>
>>>
>>> 
>>> From: Tony Kurc 
>>> Sent: Tuesday, September 3, 2019 8:33 PM
>>> To: dev@nifi.apache.org
>>> Subject: Re: [VOTE] Create NiFi Standard Libraries sub-project
>>>
>>> +1 (binding)
>>>
>>> On Wed, Sep 4, 2019 at 12:29 AM Aldrin Piri 
>> wrote:
>>>
 +1, binding

 On Tue, Sep 3, 2019 at 19:46 Yolanda Davis 
 wrote:

> +1 Create NiFi Standard Libraries (binding)
>
> On Tue, Sep 3, 2019 at 7:03 PM Koji Kawamura
> 
> wrote:
>
>> +1 Create NiFi Standard Libraries (binding)
>>
>> On Wed, Sep 4, 2019 at 7:25 AM Mike Thomsen
>> 
>> wrote:
>>>
>>> +1 binding
>>>
>>> On Tue, Sep 3, 2019 at 5:33 PM Andy LoPresto
>>> 
>> wrote:
>>>
 +1, create NiFi Standard Libraries (binding)

 Andy LoPresto
 alopre...@apache.org
 alopresto.apa...@gmail.com
 PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D
 EF69

> On Sep 3, 2019, at 2:16 PM, Bryan Bende 
 wrote:
>
> All,
>
> In a previous thread there was a plan discussed to
> restructure
 some
>> of
> the repositories in order to address several different
> issues,
 such
>> as
> build time, reusability of code, and eventually separating
> how
 the
> framework and extensions are released [1][2].
>
> The overall plan requires many steps to get there, so I'd
> like to propose starting with a small actionable step - the
> creation of a
> new
> sub-project called NiFi Standard Libraries (formerly
> referred to
 as
> nifi-commons).
>
> Project Name: Apache NiFi Standard Libraries Git Repository:
> nifi-standard-libraries
> JIRA: NIFILIBS
>
> Description:
>
> A collection of standard implementations used across the
> NiFi
>> ecosystem.
>
> Candidate Libraries:
>
> In general, each library may consist of multiple Maven
> modules,
 and
> should be independent from the rest of the ecosystem, and
> from
> other
> libraries within NiFi Standard Libraries.
>
> In addition, each library may make it's own decision about
 whether
> it
> is considered a public facing extension point/API, or an
> internal library that may be changed at any time. This
> should be
 documented
> in
> a README at the root of each library, such as
> nifi-standard-libraries/nifi-xyz/README.
>
> An initial library that has been discussed was referred to
> as 'nifi-security' and would centralize much of the security
> related
>> code
> shared by NiFi and NiFi Registry, such as shared security
> APIs,
 and
> implementations for various providers, such as
>>> LDAP/Kerberos/etc.
>
> A second candidate library would be an optimistic-locking
> library based on NiFi's revision concept. Currently this has
> been created inside nifi-registry for now [3], but could be
> moved as soon as nifi-standard-libraries exists.
>
> (This list does not have to be final in order to decide if
> we are creating NiFi Standard Libraries or not)
>
> Integration & Usage:
>
> Once NiFi Standard Libraries is created, the community can
> start creating and/or moving code there and perform releases
> as
> necessary.
>> A
> release will consist of the standard Apache source release,
> plus artifacts released to Maven central. The community can
> then
 decide
> when it is appropriate to integrate these released libraries
> into
> o

Re: [DISCUSS] rename master branch, look through code for other related issues

[Brandon DeVries]

"Instead of banning every word out
there, we should make the mental effort to do better than the
political correctness movement that stops at the surface."

I think this is a pretty clear false dichotomy.  Doing one doesn't preclude
the other.  The statement above goes on:

"So, let’s call it master-slave, and instead make a call for US, where
a sizable black population is very poor, to have free healthcare, to
have cops that are less biased against non-white people, to stop death
penalty. This makes really a difference."

... those things seem outside the direct control of the ASF.  We should try
to improve the things we can when we can.

+1 on the change

On Thu, Jun 18, 2020 at 11:24 AM Jon Logan  wrote:

> I think this blog post from the founder of Redis sums things up:
> http://antirez.com/news/122
>
> From the post:
>
> For example if I’m terminally ill, the “short living request”
> terminology may be offensive, it reminds me that I’m going to die, or
> that my father is going to die. Instead of banning every word out
> there, we should make the mental effort to do better than the
> political correctness movement that stops at the surface.
>
>
> On Thu, Jun 18, 2020 at 11:05 AM Edward Armes 
> wrote:
>
> > I agree with this, and maybe that is the potential the step forward here
> > is: issue a statement is issued saying something like this is a complex
> > issue and instead of making changes that could cause further division
> > within the community we are looking for those that are interested to help
> > form a constructive working group that will help influence and resolve
> all
> > of these issues in a positive way for all not only for project but also
> > within the wider group of apache projects.
> >
> > Edward
> >
> >
> >
> > On Thu, 18 Jun 2020, 11:17 u...@moosheimer.com, 
> wrote:
> >
> > > Language is always changing and the meaning of words is changing,
> > > sometimes positively and sometimes negatively.
> > > I think that now is time for change again and we should discuss the use
> > > of phrases and meanings.
> > >
> > > Of course we should change "Master Branch" to "Main Branch".
> > > But I also think that we shouldn't just make quick changes because it's
> > > opportune and hastily change a few words.
> > >
> > > An example: We could change Master/Slave to Leader/Follower. This may
> be
> > > a perfect choice for most people in the world.
> > > In German Leader is the English word for "Führer". And it is precisely
> > > this word that we in Germany do not actually want to use for it.
> > >
> > > What I mean is that every country and every group (e.g. religion etc.)
> > > has its own history and certain words or phrases are just not a perfect
> > > choice.
> > > We should try to go the ethically correct way worldwide.
> > >
> > > This concerns the adaptation of current words and phrases with a view
> to
> > > all: in English, Indian, Chinese, German etc. but also for indigenous
> > > peoples, different religions etc.
> > > And cultural differences should also be taken into account.
> > >
> > > What I would wish for:
> > > Apache.org should set up an "Ethics Board". A group of people of
> > > different genders, all colors, religions and from different countries
> > > and cultures all over our world.
> > > This Ethics Board should find good and for no one discriminating words
> > > or phrases for all the areas that stand out today as offensive.
> > >
> > > And it would be nice if not only computer scientists participated, but
> > > also ethicists, philosophers, engineers, various religious people,
> > > chemists, biologists, physicists, sociologists, etc.
> > >
> > > And this Council should set binding targets for all projects.
> > >
> > > Am 18.06.2020 um 09:36 schrieb Pierre Villard:
> > > >> In my perspective this should be an issue for the entire community.
> > > Being
> > > >> able to identify an issue that directly affects another person but
> not
> > > >> one’s self is the definition of privilege. If I can look at how the
> > use
> > > of
> > > >> these words in someone’s daily life or career impacts them
> negatively,
> > > > when
> > > >> the change would not harm me at all, I see that as a failure on my
> > > part. I
> > > >> understand the desire to hear from the silent majority, but active
> > > >> participation and discussion on the mailing list is the exact
> measure
> > > >> described by the Apache process for participation in the community.
> > > Those
> > > >> who speak here are the ones who will have a voice.
> > > > I could not agree more with the above.
> > > >
> > > > Le jeu. 18 juin 2020 à 04:29, Tony Kurc  a écrit :
> > > >
> > > >> I suppose I was a bit remiss in not unwinding and/or summarizing
> some
> > of
> > > >> what was in that yetus thread to prime the discussion, but a some of
> > > what
> > > >> Andy is mentioning is expanded on a bit in this ietf document [1],
> > > which is
> > > >> linked in one of the articles.
> > > >>
> > > >> 1. https://tools.ietf

Provenance repo corruption

[Brandon DeVries]

Devs,

I just opened a ticket to address an issue we've encountered with
Provenance repo corruption[1].  This would address (as is currently
partially being done) how to recover from a corrupt provenance repo.
However, the question is whether we can avoid this sort of corruption in
the first place.  The immediate thought that jumped to mind was wrapping
the writes to lucene with a write ahead log.  Obviously, this would
increase the overhead on something that is already fairly expensive.
However, in cases where provenance is *really* important, it might be worth
considering.  This could potentially be another flavor of
ProvenanceEventRepository, e.g.  WriteAheadPersistentProvenanceRepository
or FaultTolerantProvenanceRepository.  Does anyone have any thoughts /
opinions?

Brandon

[1] https://issues.apache.org/jira/browse/NIFI-2890

Re: OCSP validation not happening for cluster servers

[Brandon DeVries]

Matt,

I think the issue isn't going through the REST api.  It's that nodes of a
cluster can connect to the cluster, whether or not their certificate has
been revoked.  In other words, not a rogue random client, but a rouge nifi
node...

Brandon

On Mon, Dec 19, 2016 at 11:22 AM Matt Gilman 
wrote:

> Joe,
>
> If a server connects through the REST API it should be subject to the same
> checks as a regular user. Can you provide more details regarding the
> requests that aren't being checked correctly?
>
> Additionally, there was some discussion whether we need the additional
> checks in the first place as we may be able to leverage checks built into
> Java [1].
>
> Matt
>
> [1] https://issues.apache.org/jira/browse/NIFI-1364
>
> On Mon, Dec 19, 2016 at 10:57 AM, Joe Skora  wrote:
>
> > This could very soon be a show stopper for us.
> >
> > Does anyone have any thoughts that might help us get this straight?
> >
> > On Wed, Dec 14, 2016 at 2:23 PM, Joe Skora  wrote:
> >
> > > Running Apache NiFi 0.7.1, we see clients rejected due to OCSP
> revocation
> > > of their certificates but we think we are seeing instances where
> servers
> > > using OCSP revoked certificates are still able to connect to a cluster.
> > >
> > > Should OCSP revocation cause these servers to be rejected by the
> cluster?
> > >
> > > Could this be a configuration problem even though the revoked clients
> > > certificates are rejected?
> > >
> > > Thanks,
> > > Joe
> > >
> >
>

Re: [DISCUSS] Run Once scheduling

[Brandon DeVries]

I think answering Joe's question is step one.  However, I was curious and
tried something:

public void onTrigger(...){
  if(!isSheduled()){
return;
  }
  FlowFile flowFile = session.get()
  if (flowFile == null){
return;
  }
  session.transfer(flowFile, REL_SUCCESS);
  updateScheduledFalse();
}

This processes one FlowFile per "scheduling".  I.e., one FlowFile goes
through, and you need to stop / start to get another.  I'm not 100% that
nothing else would ever call the "built in" updateScheduled* methods, but
worst case the processor could maintain its own flag.  Also, for what it's
worth, calling updateScheduledFalse() doesn't "stop" the processor on the
graph... as Oleg mentions, this (or something like this) could potentially
be visually confusing.

I'm not sure how this fits in a production system, but this +
GenerateFlowFile and some backpressure seems possibly useful for
debugging.  I know I've faked this behavior with a GenerateFlowFile w/ run
schedule "1 day" or something before...  then again, maybe it would be best
to not create something that could be confusing / misused in a production
system.

Brandon




On Thu, Jan 12, 2017 at 1:02 PM Joe Witt  wrote:

> Naz,
>
> Why not just leave all the processes running?  If the data only
> arrives periodically that is ok, right?
>
> Thanks
> Joe
>
> On Thu, Jan 12, 2017 at 10:54 AM, Irizarry Jr., Nazario 
> wrote:
> > On a project that I am on we have been looking at using NiFi for
> orchestrations that are invoked infrequently.  For example, once a month a
> new data input product becomes available and then one wants to run it
> through a set of processing steps that can be nicely implemented using NiFi
> processors.  However, using the interval or cron scheduling for this
> purpose begins to get cumbersome after a while with the need to start and
> manually stop these occasional flows.
> >
> > It would be fairly easy to add an additional scheduling option - “Run
> Once” for this use case.  The behavior would be that when a processor is
> set to run once it automatically stops after it has successfully processed
> one input.
> >
> > What do people think?  We are willing to implement this small
> enhancement.
> >
> > Cheers,
> >
> > Naz Irizarry
> > MITRE Corp.
> > 617-893-0074 <(617)%20893-0074>
> >
> >
> >
>

Re: [VOTE] Establish Registry, a sub-project of Apache NiFi

[Brandon DeVries]

+1 binding
On Fri, Feb 10, 2017 at 6:36 PM Andre  wrote:

> +1 binding
>
> On Sat, Feb 11, 2017 at 3:40 AM, Bryan Bende  wrote:
>
> > All,
> >
> > Following a solid discussion for the past few days [1] regarding the
> > establishment of Registry as a sub-project of Apache NiFi, I'd like to
> > call a formal vote to record this important community decision and
> > establish consensus.
> >
> > The scope of this project is to define APIs for interacting with
> > resources that one or more NiFi instances may be interested in, such
> > as a flow registry for versioned flows, an extension registry for
> > extensions, and possibly other configuration resources in the future.
> > In addition, this project will provide reference implementations of
> > these registries, with the goal of allowing the community to build a
> > diverse set of implementations, such as a Git provider for versioned
> > flows, or a bintray provider for an extension registry.
> >
> > I am a +1 and looking forward to the future work in this area.
> >
> > The vote will be open for 72 hours and be a majority rule vote.
> >
> > [ ] +1 Establish Registry, a subproject of Apache NiFi
> > [ ]   0 Do not care
> > [ ]  -1 Do not establish Registry, a subproject of Apache NiFi
> >
> > Thanks,
> >
> > Bryan
> >
> > [1]
> http://mail-archives.apache.org/mod_mbox/nifi-dev/201702.mbox/%3CCALo_
> > M19euo2LLy0PVWmE70FzeLhQRcCtX6TC%3DqoiBVfn4zFQMA%40mail.gmail.com%3E
> >
>

Closing in on a NiFi 0.8.0 release?

[Brandon DeVries]

Team,

The only unresolved tickets against the 0.8.0 release[1] are for the
removal of code...  With that in mind, does anyone object to trying to push
for this (possibly final) 0.x release?

Brandon

[1]
https://issues.apache.org/jira/issues/?jql=project%20%3D%20NIFI%20AND%20fixVersion%20%3D%200.8.0%20AND%20resolution%20%3D%20Unresolved%20ORDER%20BY%20due%20ASC%2C%20priority%20DESC%2C%20created%20ASC

Re: Closing in on a NiFi 0.8.0 release?

[Brandon DeVries]

Based on Jira, there are 20 tickets marked as fixed in 0.8.0 and nowhere
else in the 0.x line[1].  Highlights from these include:

   - NIFI-2890 - Provenance Repository corruption
   - NIFI-2920 - Swapped FlowFiles are not removed from content repo when a
   queue is emptied.
   - NIFI-3055 - StandardRecordWriter can throw UTFDataFormatException
   - NIFI-3424 - CLONE for 0.x - Unable to generate Provenance Event
   because FlowFile UUID is not set
   - NIFI-3230 - Get/Put JMS broken for simple ActiveMQ SSL URIs
   - NIFI-3403 - NPE in InvokeHTTP
   - NIFI-3362 - Update FlowConfiguration.xsd TimePeriod to match
   FormatUtils
   - NIFI-2861 - ControlRate should accept more than one flow file per
   execution
   - NIFI-3350 - Reduce NiFi startup time by streamlining documentation
   extraction

Are we willing to port all of the tickets from [1] to the 0.7 branch?  Or
rather, which of them would not make the cut?  There are a couple of things
on the list that seem like new features as opposed to pure bug fixes...
although I suppose the difference between a "bug fix" and an "improvement"
is somewhat in the eye of the beholder.

Ultimately, as long as there's a release covering these issues (everything
except the NIFI-2991[2] stuff) I don't particularly care what it's called.
If there are issues left out and I need to run a SNAPSHOT of some sort to
get them, then a further 0.x release doesn't help me anyway, and I'll
withdraw my suggestion.

Brandon

[1]
https://issues.apache.org/jira/issues/?jql=project%20%3D%20NIFI%20AND%20fixVersion%20%3D%200.8.0%20and%20fixVersion%20not%20in%20(0.2.1%2C%200.3.0%2C%200.4.0%2C0.4.1%2C0.5.0%2C0.5.1%2C0.6.0%2C%200.6.1%2C0.7.0%2C%200.7.1%2C%200.7.2%2C%200.7.3)%20ORDER%20BY%20due%20ASC%2C%20priority%20DESC%2C%20created%20ASC

[2] https://issues.apache.org/jira/browse/NIFI-2991


On Fri, Feb 24, 2017 at 7:49 PM Joe Witt  wrote:

> The 0.8 fixes for licensing remove a processor (gettwitter) at this point.
> That I feel requires at least minor.  But avoiding that for now and doing
> the bug fix things and doing 073 seems legit.
>
> Will wait and see if anyone else has a different interpretation on the
> intent of our one year version guidance and then update the wiki if appears
> we have consensus.
>
> Thanks
> Joe
>
> On Feb 24, 2017 7:19 PM, "Andy LoPresto"  wrote:
>
> > Especially as nothing that would be going into the 0.x release is a major
> > feature or changes compatibility (from my understanding), I would +1 the
> > 0.7.3 suggestion.
> >
> > Andy LoPresto
> > alopre...@apache.org
> > *alopresto.apa...@gmail.com *
> > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> >
> > On Feb 24, 2017, at 2:07 PM, Tony Kurc  wrote:
> >
> > I think it is probably worth clarifying the intent of the support
> language.
> > I believe the intent was to support 0.x for a year after 1.x was
> released.
> > That was how I initially read the document you mentioned. But after a
> > re-read, I'd echo your concerns about dragging old major lines along.
> >
> > Tony
> >
> > On Fri, Feb 24, 2017 at 4:12 PM, Joe Witt  wrote:
> >
> > Brandon,
> >
> > My concern is the language used when we published this "We support the
> > newest major release line (0.x, 1.x) and any previous major release
> > lines for up to one year since the last minor release (0.6.x, 1.5.y)
> > in that line" within this document [1].
> >
> > If I read that now it seems like we're saying "if we make a minor
> > release we're going to support that for up to a year" and so each time
> > we create a new minor line on a given major line it means we are
> > resetting the clock.
> >
> > I do not believe we should give old major lines, such as 0.x, the
> > ability to drag on the community indefinitely as that reads.  I
> > believe it should be that we support a given major release line for up
> > to one year one after a new major release line is provided.
> >
> > So would like to hear peoples thoughts on that.
> >
> > If an 0.8 release is to occur the items called out are things which
> > impact licensing only (specifically the no longer allowed cat-x json
> > library). I would be far more comfortable with 0.7.3 release which
> > would be fixing whatever bugs have been addressed.  That avoids the
> > concern I noted above for this case though i'd still like us to
> > clarify that language/intent anyway.
> >
> > Thanks
> > Joe
> >
> > On Fri, Feb 24, 2017 at 3:21 PM, Brandon DeVries  wrote:
> >
> > Team,
> >
> > The only unresolved tickets against the 0.8.0 release[1] are for the
> > removal of code...  With that in mind, does anyone object to trying to
> >
> > push
> >
> > for this (possibly final) 0.x release?
> >
> > Brandon
> >
> > [1]
> > https://issues.apache.org/jira/issues/?jql=project%20%
> >
> > 3D%20NIFI%20AND%20fixVersion%20%3D%200.8.0%20AND%20resolution%20%3D%
> > 20Unresolved%20ORDER%20BY%20due%20ASC%2C%20priority%
> > 20DESC%2C%20created%20ASC
> >
> >
> >
>

Re: RTC clarification

[Brandon DeVries]

There are always exceptions, but I think the best way to ensure that the
spirit of what we're going for is being followed is to say "no one commits
their own code".  While additional eyes are never going to be a bad thing,
requiring a second person to "sign off" on and then commit the code would
seem to help keep the writing and reviewing steps separate, as we want them
to be.

Secondarily, with no judgement attached to the statement, a review from
someone who's learning the code base might not have the same depth of
understanding and awareness of context as a review from someone with more
experience on the project.  I would think for a review to be trusted, we
would need a certain level of trust in the reviewer... and the best current
process we have for that is committer status.  Like James said, "If we knew
the reviewer well enough to accept their judgement, why not make them a
committer?"

Brandon

On Fri, Jul 7, 2017 at 9:24 AM Bryan Bende  wrote:

> I agree with encouraging reviews from everyone, but I lean towards
> "binding" reviews coming from committers.
>
> If we allow any review to be binding, there could be completely
> different levels of review that occur...
>
> There could be someone who isn't a committer yet, but has been
> contributing already and will probably do a very thorough review of
> someone's contribution, and there could be someone else who we never
> interacted with us before and writes "+1 LGTM" on a PR and we have no
> idea if they know what they are talking about or if they even tried
> the contribution to see if it works. Obviously a committer can also
> write "+1 LGTM", but I think when that comes from a committer it holds
> more weight.
>
> I think we may also want to clarify if we are only talking about
> "submitted by committer, reviewed by non-committer" or also talking
> about "submitted by non-committer, reviewed by non-committer".
>
> For the first case I can see the argument that since the contribution
> is from a committer who is already trusted, they can make the right
> judgement call based on the review. Where as in the second case, just
> because a community member submitted something and another community
> member says it looks good, doesn't necessarily mean a committer should
> come along and automatically merge it in.
>
>
> On Fri, Jul 7, 2017 at 4:13 AM, Michael Hogue
>  wrote:
> > Thanks for fielding the question, Tony.
> >
> > Joe and James' statements both make sense. I suppose a case by case
> > analysis could be carried out, too. For example, since I'm mostly
> > unfamiliar with the code base but am looking to gain familiarity, I'm
> > reviewing pretty straightforward or trivial PRs. My plan was to continue
> > doing that until I felt comfortable reviewing something with a larger
> > impact, such as the new TCPListenRecord processor implementation [1].
> > However, as Tony explained, my original question was whether that sort of
> > review would be binding or whether I should be doing it at all. I think
> > both of those questions were answered here in that ultimately committer
> > sign off is needed, but reviews may be binding regardless of source.
> >
> > Thanks for the feedback!
> >
> > Mike
> >
> >
> > [1] https://github.com/apache/nifi/pull/1987
> > On Fri, Jul 7, 2017 at 01:14 James Wing  wrote:
> >
> >> We should definitely encourage review feedback from non-committers.
> >> Getting additional perspectives, interest, and enthusiasm from users is
> >> critical for any project, doubly so for an integrating application where
> >> committers cannot be experts in all the systems we are integrating
> with.  I
> >> believe NiFi could use more review bandwidth.  Are missing out on
> potential
> >> reviewers because of the current policy?
> >>
> >> I do not have any experience with non-committer "binding reviews" as
> >> described in the Apache Gossip thread.  How would that work?  Wouldn't a
> >> committer have to review the review and decide to commit?  If we knew
> the
> >> reviewer well enough to accept their judgement, why not make them a
> >> committer?
> >>
> >> My expectation is that many non-committer reviews are helpful and
> >> constructive, but not necessarily 100% comprehensive.  Reviewers might
> >> comment on the JIRA ticket without working with the PR, or try the
> proposed
> >> change without reviewing the code, tests, etc.  All great stuff, but
> >> backstopped by committers.
> >>
> >> Thanks,
> >>
> >> James
> >>
> >> On Thu, Jul 6, 2017 at 7:30 PM, Joe Witt  wrote:
> >>
> >> > It is undefined at this point and I agree we should reach consensus
> >> > and document it.
> >> >
> >> > I am in favor making non-committer reviews binding.
> >> >
> >> > Why do we do RTC:
> >> > - To help bring along new committers/grow the community
> >> > - To help promote quality by having peer reviews
> >> >
> >> > Enabling non-committer reviews to be binding still allows both of
> >> > those to be true.
> >> >
> >> > On Thu, Jul 6, 2017 at 10:10 PM, Tony

Re: [DISCUSS} Closing in on a NiFi 1.4.0 release?

[Brandon DeVries]

+1, it seems like we're do for a release.  It's been a week, and a couple
of the mentioned tickets have shown progress... but a number haven't.  If
no one wants to get them wrapped up, can we put them off to 1.5?

On Mon, Sep 11, 2017 at 11:39 AM Wes Lawrence  wrote:

> I'd also like NIFI-4242 and NIFI-4181 added (or if it's OK, I can label
> their fix versions for 1.4.0. I wasn't sure sure I could do that, or if
> that should be left to PMC/Commiters)
>
> NIFI-4242  has a patch
> with feedback, and I'll be pushing a new patch addressing the feedback
> later today.
> NIFI-4181  has a patch
> with feedback, and I'd like to address the feedback for 1.4.0 if possible.
>
> --Wes
>
> On Mon, Sep 11, 2017 at 10:33 AM, Christopher Currie <
> christop...@currie.com
> > wrote:
>
> > In addition to this list, I'd like to request that
> > https://issues.apache.org/jira/browse/NIFI-3950 also be added. I
> created a
> > PR for it, and I'm happy to work with a committer to clean it up for
> > project standards if needed.
> >
> > Christopher
> > On Mon, Sep 11, 2017 at 6:44 AM James Wing  wrote:
> >
> > > +1  It does seem to be about time to get a release out, and we should
> > > certainly take advantage of your offer to performing RM duties.  Thank
> > you
> > > for that.
> > >
> > > On Sun, Sep 10, 2017 at 5:45 PM, Jeff  wrote:
> > >
> > > > All,
> > > >
> > > > On the dev and users lists there have been some questions about when
> > the
> > > > next release of NiFi would be.  I would like perform RM duties, and
> get
> > > > things started on identifying what the community would like to
> include
> > in
> > > > the release of NiFi 1.4.0 that has not already been reviewed and
> > > committed
> > > > to master.
> > > >
> > > > There are 11 unresolved JIRAs tagged with a fix version of 1.4.0.
> > > >
> > > > https://issues.apache.org/jira/issues/?jql=project%20%
> > > > 3D%20NIFI%20AND%20resolution%20%3D%20Unresolved%20AND%
> > > > 20fixVersion%20%3D%201.4.0%20ORDER%20BY%20priority%
> > 20DESC%2C%20key%20DESC
> > > >
> > > > Are there any reasons to hold off on a 1.4.0 release?  Are there
> > > particular
> > > > JIRAs that the community considers necessary to have as part of the
> > > > release? Let's discuss!
> > > >
> > > > Thanks,
> > > > Jeff
> > > >
> > >
> >
>

Re: [DISCUSS} Closing in on a NiFi 1.4.0 release?

[Brandon DeVries]

There are significant changes in 1.4.0 that I am actively waiting on...

On Mon, Sep 18, 2017 at 2:25 PM Russell Bateman 
wrote:

> I don't know. Are we due for a release? Is time-since the significant
> factor in a release cycle or is growing features part of it?
>
> 1.3.0 subsists with no bump of the third digit. This is an oddly stable
> .0 product (though the third digit had somewhat different semantics in
> NiFi 0.x). No bug fixes to 1.3.0 in its roughly 6-month history? That's
> an achievement.
>
> If there have been important features worked on and ready to go, by all
> means, let's have 1.4.0. But if 1.4.0 is little more than 1.3.1, let's
> rethink why we'd do that.
>
> Russ
>
>
> On 09/18/2017 12:08 PM, Brandon DeVries wrote:
> > +1, it seems like we're do for a release.  It's been a week, and a couple
> > of the mentioned tickets have shown progress... but a number haven't.  If
> > no one wants to get them wrapped up, can we put them off to 1.5?
> >
> > On Mon, Sep 11, 2017 at 11:39 AM Wes Lawrence 
> wrote:
> >
> >> I'd also like NIFI-4242 and NIFI-4181 added (or if it's OK, I can label
> >> their fix versions for 1.4.0. I wasn't sure sure I could do that, or if
> >> that should be left to PMC/Commiters)
> >>
> >> NIFI-4242 <https://issues.apache.org/jira/browse/NIFI-4242> has a patch
> >> with feedback, and I'll be pushing a new patch addressing the feedback
> >> later today.
> >> NIFI-4181 <https://issues.apache.org/jira/browse/NIFI-4181> has a patch
> >> with feedback, and I'd like to address the feedback for 1.4.0 if
> possible.
> >>
> >> --Wes
> >>
> >> On Mon, Sep 11, 2017 at 10:33 AM, Christopher Currie <
> >> christop...@currie.com
> >>> wrote:
> >>> In addition to this list, I'd like to request that
> >>> https://issues.apache.org/jira/browse/NIFI-3950 also be added. I
> >> created a
> >>> PR for it, and I'm happy to work with a committer to clean it up for
> >>> project standards if needed.
> >>>
> >>> Christopher
> >>> On Mon, Sep 11, 2017 at 6:44 AM James Wing  wrote:
> >>>
> >>>> +1  It does seem to be about time to get a release out, and we should
> >>>> certainly take advantage of your offer to performing RM duties.  Thank
> >>> you
> >>>> for that.
> >>>>
> >>>> On Sun, Sep 10, 2017 at 5:45 PM, Jeff  wrote:
> >>>>
> >>>>> All,
> >>>>>
> >>>>> On the dev and users lists there have been some questions about when
> >>> the
> >>>>> next release of NiFi would be.  I would like perform RM duties, and
> >> get
> >>>>> things started on identifying what the community would like to
> >> include
> >>> in
> >>>>> the release of NiFi 1.4.0 that has not already been reviewed and
> >>>> committed
> >>>>> to master.
> >>>>>
> >>>>> There are 11 unresolved JIRAs tagged with a fix version of 1.4.0.
> >>>>>
> >>>>> https://issues.apache.org/jira/issues/?jql=project%20%
> >>>>> 3D%20NIFI%20AND%20resolution%20%3D%20Unresolved%20AND%
> >>>>> 20fixVersion%20%3D%201.4.0%20ORDER%20BY%20priority%
> >>> 20DESC%2C%20key%20DESC
> >>>>> Are there any reasons to hold off on a 1.4.0 release?  Are there
> >>>> particular
> >>>>> JIRAs that the community considers necessary to have as part of the
> >>>>> release? Let's discuss!
> >>>>>
> >>>>> Thanks,
> >>>>> Jeff
> >>>>>
>
>

Re: [EXT] Re: [DISCUSS} Closing in on a NiFi 1.4.0 release?

[Brandon DeVries]

All,

I think we should plan on calling for a vote on Friday.  That gives two
days to wrap up any outstanding tickets that anyone feels really belong in
1.4.  At that point the remaining tickets can be shifted to a future
release.

If there are tickets that are not getting the attention they need to make
it into the release, let the list know.

Any objections?

Brandon

On Wed, Sep 20, 2017 at 12:32 AM Koji Kawamura 
wrote:

> Hi Paul,
>
> I was able to reproduce the GenerateTableFetch processor issue
> reported by NIFI-4395.
> Please go ahead and provide a PR, I can review it.
>
> Thanks,
> Koji
>
> On Wed, Sep 20, 2017 at 1:10 PM, Paul Gibeault (pagibeault)
>  wrote:
> > We have submitted this JIRA ticket:
> > https://issues.apache.org/jira/browse/NIFI-4395
> >
> > This issue causes GenerateTableFetch processor to malfunction after a
> server restart.
> >
> > We are very interested in getting this released in 1.4.0 and are willing
> to provide the PR if there is still time.
> >
> > Thanks,
> > Paul Gibeault
> >
> >
> > -Original Message-
> > From: Michael Hogue [mailto:michael.p.hogu...@gmail.com]
> > Sent: Tuesday, September 19, 2017 9:36 AM
> > To: dev@nifi.apache.org
> > Subject: [EXT] Re: [DISCUSS} Closing in on a NiFi 1.4.0 release?
> >
> > All,
> >
> >There are a couple of issues with open PRs that i think would be
> desirable to get into 1.4.0:
> >
> >   - https://github.com/apache/nifi/pull/2163 - trivial one-liner in
> ListenGRPC
> >   - https://github.com/apache/nifi/pull/1985 - support TLS algorithm
> selection via SSLContextService in HandleHTTPRequest
> >
> > Thanks,
> > Mike
> >
> > On Tue, Sep 19, 2017 at 10:46 AM Mark Bean 
> wrote:
> >
> >> I agree with including only those which can be completed quickly in
> 1.4.0.
> >> We are anxious for the next release to begin exercising some of the
> >> new features. IMO it's time to get 1.4.0 out the door.
> >>
> >> Thanks,
> >> Mark
> >>
> >> On Mon, Sep 18, 2017 at 6:59 PM, Jeff  wrote:
> >>
> >> > Still good.  Was looking through tickets yesterday and today and
> >> > while review progress has been made on some PRs, it might be best to
> >> > move JIRAs tagged for 1.4.0 that have PRs and aren't on the cusp of
> >> > being committed
> >> to
> >> > post 1.4.0.  Thoughts?
> >> >
> >> > On Mon, Sep 18, 2017 at 2:50 PM Joe Witt  wrote:
> >> >
> >> > > Definitely agree with Brandon that due for a 1.4.0 and it has some
> >> > > really nice things in it
> >> > >
> >> > > Jeff Storck volunteered to RM.  Jeff you still good?  Anything I
> >> > > can
> >> help
> >> > > with?
> >> > >
> >> > >
> >> > > Thanks
> >> > > Joe
> >> > >
> >> > > On Mon, Sep 18, 2017 at 2:28 PM, Brandon DeVries 
> wrote:
> >> > > > There are significant changes in 1.4.0 that I am actively
> >> > > > waiting
> >> on...
> >> > > >
> >> > > > On Mon, Sep 18, 2017 at 2:25 PM Russell Bateman <
> >> r...@windofkeltia.com
> >> > >
> >> > > > wrote:
> >> > > >
> >> > > >> I don't know. Are we due for a release? Is time-since the
> >> significant
> >> > > >> factor in a release cycle or is growing features part of it?
> >> > > >>
> >> > > >> 1.3.0 subsists with no bump of the third digit. This is an
> >> > > >> oddly
> >> > stable
> >> > > >> .0 product (though the third digit had somewhat different
> >> > > >> semantics
> >> in
> >> > > >> NiFi 0.x). No bug fixes to 1.3.0 in its roughly 6-month history?
> >> > That's
> >> > > >> an achievement.
> >> > > >>
> >> > > >> If there have been important features worked on and ready to
> >> > > >> go, by
> >> > all
> >> > > >> means, let's have 1.4.0. But if 1.4.0 is little more than
> >> > > >> 1.3.1,
> >> let's
> >> > > >> rethink why we'd do that.
> >> > > >>
> >> > > >> Russ
> >> > > >>
> >> > > >>
&g

Re: [EXT] Re: [DISCUSS} Closing in on a NiFi 1.4.0 release?

[Brandon DeVries]

Mayank,

Try this:

https://issues.apache.org/jira/issues/?jql=project%20%3D%20NIFI%20AND%20fixVersion%20%3D%201.4.0%20ORDER%20BY%20status%20DESC

Brandon


On Wed, Sep 20, 2017 at 9:57 AM mayank rathi  wrote:

> Hello All,
>
> How can we find out list of fixes that will go in 1.4.0 release?
>
> Thanks!!
>
> On Wed, Sep 20, 2017 at 9:53 AM, Brandon DeVries  wrote:
>
> > All,
> >
> > I think we should plan on calling for a vote on Friday.  That gives two
> > days to wrap up any outstanding tickets that anyone feels really belong
> in
> > 1.4.  At that point the remaining tickets can be shifted to a future
> > release.
> >
> > If there are tickets that are not getting the attention they need to make
> > it into the release, let the list know.
> >
> > Any objections?
> >
> > Brandon
> >
> > On Wed, Sep 20, 2017 at 12:32 AM Koji Kawamura 
> > wrote:
> >
> > > Hi Paul,
> > >
> > > I was able to reproduce the GenerateTableFetch processor issue
> > > reported by NIFI-4395.
> > > Please go ahead and provide a PR, I can review it.
> > >
> > > Thanks,
> > > Koji
> > >
> > > On Wed, Sep 20, 2017 at 1:10 PM, Paul Gibeault (pagibeault)
> > >  wrote:
> > > > We have submitted this JIRA ticket:
> > > > https://issues.apache.org/jira/browse/NIFI-4395
> > > >
> > > > This issue causes GenerateTableFetch processor to malfunction after a
> > > server restart.
> > > >
> > > > We are very interested in getting this released in 1.4.0 and are
> > willing
> > > to provide the PR if there is still time.
> > > >
> > > > Thanks,
> > > > Paul Gibeault
> > > >
> > > >
> > > > -Original Message-
> > > > From: Michael Hogue [mailto:michael.p.hogu...@gmail.com]
> > > > Sent: Tuesday, September 19, 2017 9:36 AM
> > > > To: dev@nifi.apache.org
> > > > Subject: [EXT] Re: [DISCUSS} Closing in on a NiFi 1.4.0 release?
> > > >
> > > > All,
> > > >
> > > >There are a couple of issues with open PRs that i think would be
> > > desirable to get into 1.4.0:
> > > >
> > > >   - https://github.com/apache/nifi/pull/2163 - trivial one-liner in
> > > ListenGRPC
> > > >   - https://github.com/apache/nifi/pull/1985 - support TLS algorithm
> > > selection via SSLContextService in HandleHTTPRequest
> > > >
> > > > Thanks,
> > > > Mike
> > > >
> > > > On Tue, Sep 19, 2017 at 10:46 AM Mark Bean 
> > > wrote:
> > > >
> > > >> I agree with including only those which can be completed quickly in
> > > 1.4.0.
> > > >> We are anxious for the next release to begin exercising some of the
> > > >> new features. IMO it's time to get 1.4.0 out the door.
> > > >>
> > > >> Thanks,
> > > >> Mark
> > > >>
> > > >> On Mon, Sep 18, 2017 at 6:59 PM, Jeff  wrote:
> > > >>
> > > >> > Still good.  Was looking through tickets yesterday and today and
> > > >> > while review progress has been made on some PRs, it might be best
> to
> > > >> > move JIRAs tagged for 1.4.0 that have PRs and aren't on the cusp
> of
> > > >> > being committed
> > > >> to
> > > >> > post 1.4.0.  Thoughts?
> > > >> >
> > > >> > On Mon, Sep 18, 2017 at 2:50 PM Joe Witt 
> > wrote:
> > > >> >
> > > >> > > Definitely agree with Brandon that due for a 1.4.0 and it has
> some
> > > >> > > really nice things in it
> > > >> > >
> > > >> > > Jeff Storck volunteered to RM.  Jeff you still good?  Anything I
> > > >> > > can
> > > >> help
> > > >> > > with?
> > > >> > >
> > > >> > >
> > > >> > > Thanks
> > > >> > > Joe
> > > >> > >
> > > >> > > On Mon, Sep 18, 2017 at 2:28 PM, Brandon DeVries 
> > > wrote:
> > > >> > > > There are significant changes in 1.4.0 that I am actively
> > > >> > > > waiting
> > > >> on...
> > > >> > > >
> > > >> > > > On Mon, Sep 18, 2017 at 2:25 P

Re: [EXT] Re: [DISCUSS} Closing in on a NiFi 1.4.0 release?

[Brandon DeVries]

Milan,

 So that Jeff has time to create the RC, I'd say 5 pm EST today (9/21).

Brandon

On Thu, Sep 21, 2017 at 4:18 AM Milan Chandna
 wrote:

> Do we have a deadline to get the issue resolved, to include it in 1.4.0
> release.
>
> -Original Message-
> From: Adam Taft [mailto:a...@adamtaft.com]
> Sent: Thursday, September 21, 2017 9:20 AM
> To: dev@nifi.apache.org
> Subject: Re: [EXT] Re: [DISCUSS} Closing in on a NiFi 1.4.0 release?
>
> Here's another good link to try, maybe a little easier to read:
>
>
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fprojects%2FNIFI%2Fversions%2F12340589&data=02%7C01%7CMilan.Chandna%40microsoft.com%7C21581ed9a39548be836e08d500a3e900%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636415626344144936&sdata=kO%2ByQJA%2FchMrvYYMpUZ%2FGOj%2FYVNEWgbmtahAr6ITrVk%3D&reserved=0
>
>
>
> On Wed, Sep 20, 2017 at 8:01 AM, Brandon DeVries  wrote:
>
> > Mayank,
> >
> > Try this:
> >
> > https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissue
> > s.apache.org%2Fjira%2Fissues%2F%3Fjql%3Dproject%2520%25&data=02%7C01%7
> > CMilan.Chandna%40microsoft.com%7C21581ed9a39548be836e08d500a3e900%7C72
> > f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636415626344144936&sdata=U5dI
> > A%2FD0S3yLy4trFBag8IZ3mVJtCD1tWtdLc2hZ2rQ%3D&reserved=0
> > 3D%20NIFI%20AND%20fixVersion%20%3D%201.4.0%20ORDER%20BY%20status%20DES
> > C
> >
> > Brandon
> >
> >
> > On Wed, Sep 20, 2017 at 9:57 AM mayank rathi 
> > wrote:
> >
> > > Hello All,
> > >
> > > How can we find out list of fixes that will go in 1.4.0 release?
> > >
> > > Thanks!!
> > >
> > > On Wed, Sep 20, 2017 at 9:53 AM, Brandon DeVries  wrote:
> > >
> > > > All,
> > > >
> > > > I think we should plan on calling for a vote on Friday.  That
> > > > gives two days to wrap up any outstanding tickets that anyone
> > > > feels really belong
> > > in
> > > > 1.4.  At that point the remaining tickets can be shifted to a
> > > > future release.
> > > >
> > > > If there are tickets that are not getting the attention they need
> > > > to
> > make
> > > > it into the release, let the list know.
> > > >
> > > > Any objections?
> > > >
> > > > Brandon
> > > >
> > > > On Wed, Sep 20, 2017 at 12:32 AM Koji Kawamura
> > > >  > >
> > > > wrote:
> > > >
> > > > > Hi Paul,
> > > > >
> > > > > I was able to reproduce the GenerateTableFetch processor issue
> > > > > reported by NIFI-4395.
> > > > > Please go ahead and provide a PR, I can review it.
> > > > >
> > > > > Thanks,
> > > > > Koji
> > > > >
> > > > > On Wed, Sep 20, 2017 at 1:10 PM, Paul Gibeault (pagibeault)
> > > > >  wrote:
> > > > > > We have submitted this JIRA ticket:
> > > > > > https://na01.safelinks.protection.outlook.com/?url=https%3A%2F
> > > > > > %2Fissues.apache.org%2Fjira%2Fbrowse%2FNIFI-4395&data=02%7C01%
> > > > > > 7CMilan.Chandna%40microsoft.com%7C21581ed9a39548be836e08d500a3
> > > > > > e900%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636415626344
> > > > > > 144936&sdata=cSKg20gPxzXXabttEoU4poDyu8k%2BkE2bFybrIc3NKSI%3D&
> > > > > > reserved=0
> > > > > >
> > > > > > This issue causes GenerateTableFetch processor to malfunction
> > after a
> > > > > server restart.
> > > > > >
> > > > > > We are very interested in getting this released in 1.4.0 and
> > > > > > are
> > > > willing
> > > > > to provide the PR if there is still time.
> > > > > >
> > > > > > Thanks,
> > > > > > Paul Gibeault
> > > > > >
> > > > > >
> > > > > > -Original Message-
> > > > > > From: Michael Hogue [mailto:michael.p.hogu...@gmail.com]
> > > > > > Sent: Tuesday, September 19, 2017 9:36 AM
> > > > > > To: dev@nifi.apache.org
> > > > > > Subject: [EXT] Re: [DISCUSS} Closing in on a NiFi 1.4.0 release?
> > > > > >
> > > > > > All,
> > > > > >
> >

Re: [CANCEL] [VOTE] Release Apache NiFi 1.4.0

[Brandon DeVries]

Jeff,

Any updates on RC2?

Brandon

On Mon, Sep 25, 2017 at 4:56 PM Jeff  wrote:

> Mark,
>
> I also would like to get RC2 out as soon as possible, but due to time
> constraints on other tasks, I will not be able to create RC2 until tomorrow
> afternoon.  However, with a 72-hour voting window resulting in binding
> votes to release RC2, 1.4.0 could be released by the end of the week.
>
> FYI, for those that would like to get a head start on using what would be
> released in 1.4.0, you can build off of the master branch and have
> essentially what will be released in RC2.  The Apache NiFi Quickstart [1]
> contains documentation on cloning the project from Github, building from
> source code, and running NiFi.
>
> [1] http://nifi.apache.org/quickstart.html
>
> On Mon, Sep 25, 2017 at 3:51 PM Mark Bean  wrote:
>
> > Jeff,
> >
> > All outstanding issues have been resolved/committed, right? Can RC2 get
> > turned around quickly? We were hoping to have an official 1.4.0 release
> by
> > the end of the week.
> >
> > Thanks,
> > Mark
> >
> >
> > On Mon, Sep 25, 2017 at 2:51 PM, Jeff  wrote:
> >
> > > Given the issues discovered today, I am canceling the voting for RC1.
> > >
> > > I will prepare RC2 when I get a bit of time to work on it, hopefully
> > within
> > > the next day or two, and start another vote thread.
> > >
> > > Here's a list of issues that will be addressed by RC2:
> > > https://issues.apache.org/jira/browse/NIFI-4345
> > > https://issues.apache.org/jira/browse/NIFI-4416
> > > https://issues.apache.org/jira/browse/NIFI-4418
> > >
> > >
> > > On Mon, Sep 25, 2017 at 1:12 PM Joe Witt  wrote:
> > >
> > > > Ok good this makes a lot more sense now.  One JIRA will fix the
> > > > erroneous duplicates.  And this JIRA, which can be done later, will
> > > > correct that we should have actually not even have noticed the
> > > > duplicates and Rick's flow should have worked correctly automatically
> > > > [1].
> > > >
> > > >
> > > > [1] https://issues.apache.org/jira/browse/NIFI-4420
> > > >
> > > > Thanks
> > > > Joe
> > > >
> > > > On Mon, Sep 25, 2017 at 12:49 PM, Bryan Bende 
> > wrote:
> > > > > I think the reason for the upgrade issue was the following...
> > > > >
> > > > > Normally there is an automatic upgrade of component versions, with
> > the
> > > > > following logic:
> > > > >
> > > > > - If the flow says you are using version X of a component, and
> during
> > > > > startup version X is not found, but version Y is found, and
> version Y
> > > > > is the only version of that component, then version Y is selected.
> > > > >
> > > > > - If the flow says you are using version X of a component, and
> during
> > > > > startup more than one version of the component is found, then we
> > can't
> > > > > automatically select one, so a ghost component would be created as
> > > > > place-holder.
> > > > >
> > > > > This is how all the components would normally go from 1.3.0 to
> 1.4.0
> > > > > on an upgrade.
> > > > >
> > > > > In Richard's flow, he was using the 1.3.0 XMLFileLookupService, and
> > > > > when he upgraded it found a 1.4.0 version from the lookup services
> > > > > NAR, and also a 1.4.0 version from the Mongo services NAR, and
> > > > > therefore fell into the second case described above.
> > > > >
> > > > > Deleting the service and re-creating it is one way to resolve the
> > > > > issue, I also believe you could go into the controller services
> table
> > > > > and select to "Change Version" on the service and select the
> version
> > > > > from the lookup services NAR.
> > > > >
> > > > >
> > > > > On Mon, Sep 25, 2017 at 12:28 PM, Matt Burgess <
> mattyb...@apache.org
> > >
> > > > wrote:
> > > > >> All,
> > > > >>
> > > > >> I verified that Joey is correct and that dependency causes the
> > > > >> duplicates. I reopened NIFI-4345 and submitted a PR.
> > > > >>
> > > > >> Regards,
> > > > >> Matt
> > > > >>
> > > > >> [1] https://issues.apache.org/jira/browse/NIFI-4345
> > > > >> [2] https://github.com/apache/nifi/pull/2174
> > > > >>
> > > > >> On Mon, Sep 25, 2017 at 11:45 AM, Richard St. John <
> > > rstjoh...@gmail.com>
> > > > wrote:
> > > > >>> Joey,
> > > > >>>
> > > > >>> That sounds like that is the issue.
> > > > >>>
> > > > >>> Rick.
> > > > >>>
> > > > >>> --
> > > > >>> Richard St. John, PhD
> > > > >>> Asymmetrik
> > > > >>> 141 National Business Pkwy, Suite 110
> > > > >>> Annapolis Junction, MD 20701
> > > > >>>
> > > > >>> On Sep 25, 2017, 11:44 AM -0400, Joey Frazee <
> > joey.fra...@icloud.com
> > > >,
> > > > wrote:
> > > >  I think there could be an issue with the deps in the
> > > > nifi-mongodb-services-nar. It includes nifi-lookup-services which
> > should
> > > > either be unnecessary or should just be provided scope (just need the
> > > > services API dependency). So it’s possible that all the impls in
> > > > nifi-lookup-services are indeed included twice.
> > > > 
> > > >  Does that jive with what you’re seeing? I.e., for LookupService
> > > >

Re: [DISCUSS] Apache NiFi distribution has grown too large

[Brandon DeVries]

I agree... Long term extension registry, short term one repo with different
assemblies (e.g. standard, slim, analytic, etc...).

Brandon

On Sat, Jan 13, 2018 at 1:35 PM Pierre Villard 
wrote:

> Option #3 also has my preference. But it's probably a good idea to only
> keep one git repo and play with the assembly and Maven profiles for the
> releases, no? It'd be certainly easier for release management process. But
> this decision could also depend on how the option #3 is going to be
> implemented I guess.
>
> 2018-01-13 6:36 GMT-07:00 Joe Witt :
>
> > thanks tony!
> >
> > On Jan 12, 2018 10:48 PM, "Tony Kurc"  wrote:
> >
> > > I put some of the data I was working with on the wiki -
> > >
> > > https://cwiki.apache.org/confluence/display/NIFI/NiFi+1.5.0+nar+files
> > >
> > > On Fri, Jan 12, 2018 at 10:28 PM, Jeremy Dyer 
> wrote:
> > >
> > > > So my favorite option is Bryan’s option number “three” of using the
> > > > extension registry. Now my thought is do we really need to add
> > complexity
> > > > and do anything in the mean time or just focus on that? Meaning we
> have
> > > > roughly 500mb of available capacity today so why don’t we spend those
> > man
> > > > hours we would spend on getting the second repo up on the extension
> > > > registry instead?
> > > >
> > > > @Bryan do you have thoughts about the deployment of those bars in the
> > > > extension registry? Since we won’t be able to build the release
> binary
> > > > anymore would we still need to create separate repos for the nars or
> > > no?? I
> > > > have used the registry a little but I’m not 100% sure on your vision
> > for
> > > > the nars
> > > >
> > > > - Jeremy Dyer
> > > >
> > > > Sent from my iPhone
> > > >
> > > > > On Jan 12, 2018, at 10:18 PM, Tony Kurc  wrote:
> > > > >
> > > > > I was looking at nar sizes, and thought some data may be helpful. I
> > > used
> > > > my recent RC1 verification as a basis for getting file sizes, and
> just
> > > got
> > > > the file size for each file in the assembly named "*.nar". I don't
> know
> > > > whether the images I pasted in will go through, but I made some
> > graphs.b
> > > > The first is a histogram of nar file size in buckets of 10MB. The
> > second
> > > > basically is similar to a cumulative distribution, the x axis is the
> > > "rank"
> > > > of the nar (smallest to largest), and the y-axis is how what fraction
> > of
> > > > the all the sizes of the nars together are that rank or lower. In
> other
> > > > words, on the graph, the dot at 60 and ~27 means that the smallest 60
> > > nars
> > > > contribute only ~27% of the total. Of note, the standard and
> framework
> > > nars
> > > > are at 83 and 84.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >> On Fri, Jan 12, 2018 at 5:04 PM, Michael Moser <
> moser...@gmail.com>
> > > > wrote:
> > > > >> And of course, as I hit  I thought of one more thing.
> > > > >>
> > > > >> We could keep all of the code in 1 git repo (1 project) but the
> > > > >> nifi-assembly part of the build could be broken up to build core
> > NiFi
> > > > >> separately from the tar/zip functional grouping of other NARs.
> > > > >>
> > > > >> On Fri, Jan 12, 2018 at 5:01 PM, Michael Moser <
> moser...@gmail.com>
> > > > wrote:
> > > > >>
> > > > >> > Long term I would also like to see #3 be the solution.  I think
> > what
> > > > >> > Joseph N described could be part of the capabilities of #3.
> > > > >> >
> > > > >> > I would like to add a note of caution with respect to
> reorganizing
> > > and
> > > > >> > releasing extension bundles separately:
> > > > >> >
> > > > >> >- the burden on release manager expands because many more
> > > projects
> > > > >> >have to be released; probably not all on each release cycle
> but
> > > it
> > > > could
> > > > >> >still be many
> > > > >> >- the chance of accidentally forgetting to release a project
> > in a
> > > > >> >release cycle becomes non-zero
> > > > >> >- sharing code between projects gets a bit harder because you
> > > have
> > > > to
> > > > >> >manage releasing projects in a specific order
> > > > >> >- it becomes harder to find all of the projects that need to
> > > change
> > > > >> >when shared code is added
> > > > >> >- the simple act of finding code becomes harder ... in which
> > > > project
> > > > >> >is that class in? (IDEs like IntelliJ can search in 1
> project,
> > > but
> > > > if they
> > > > >> >search across multiple projects, then I haven't learned how)
> > > > >> >
> > > > >> > I used to maintain several nars in separate projects, and
> recently
> > > > >> > reorganized them into 1 project (following NiFi's multi-module
> > maven
> > > > build)
> > > > >> > and life has become much easier!
> > > > >> >
> > > > >> > -- Mike
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > On Fri, Jan 12, 2018 at 4:33 PM, Chris Herrera <
> > > > chris.herrer...@gmail.com>
> > > > >> > wrote:
> > > > >> >
> > > > >> >> I very much like the solution

Re: PRC_NIFI_Pricing_Service - Build # 6 - Fixed!

[Brandon DeVries]

sweet!

On Fri, Jan 19, 2018 at 2:46 PM Michael Moser  wrote:

> Somebody's Jenkins is spamming everyone who has made a commit recently!
>
>
> -- Forwarded message --
> From: 
> Date: Fri, Jan 19, 2018 at 11:23 AM
> Subject: PRC_NIFI_Pricing_Service - Build # 6 - Fixed!
> To: tea...@bol.com, ald...@apache.org, alopre...@apache.org,
> alopresto.apa...@gmail.com, bbe...@apache.org, gardellajuanpa...@gmail.com,
> ijokaruma...@apache.org, joew...@apache.org, jsk...@apache.org,
> jtsw...@gmail.com, jvw...@gmail.com, marka...@hotmail.com,
> matt.c.gil...@gmail.com, mattyb...@apache.org, mose...@apache.org,
> phroc...@apache.org, pierre.villard...@gmail.com, rhaverm...@bol.com,
> scottyas...@gmail.com
>
>
> PRC_NIFI_Pricing_Service - Build # 6 - Fixed:
>
> Check console output at
> https://jenkins.tools.bol.com/job/PRC_NIFI_Pricing_Service/6/ to view the
> results.
>
> These are the changes incorporated in this build:
> [Ramon Havermans] OPM-1677 Jenkins times out
>
>
>
> Last 250 lines of the build log:
> [...truncated 2.53 MB...]
> [INFO] nifi-update-attribute-ui ... SUCCESS [
> 1.737 s]
> [INFO] nifi-update-attribute-nar .. SUCCESS [
> 0.814 s]
> [INFO] nifi-kafka-bundle .. SUCCESS [
> 0.016 s]
> [INFO] nifi-kafka-0-8-processors .. SUCCESS [
> 4.615 s]
> [INFO] nifi-kafka-0-9-processors .. SUCCESS [
> 5.717 s]
> [INFO] nifi-kafka-0-10-processors . SUCCESS [
> 6.107 s]
> [INFO] nifi-kafka-0-11-processors . SUCCESS [
> 6.023 s]
> [INFO] nifi-kafka-1-0-processors .. SUCCESS [
> 8.076 s]
> [INFO] nifi-kafka-0-8-nar . SUCCESS [
> 1.015 s]
> [INFO] nifi-kafka-0-9-nar . SUCCESS [
> 0.443 s]
> [INFO] nifi-kafka-0-10-nar  SUCCESS [
> 0.524 s]
> [INFO] nifi-kafka-0-11-nar  SUCCESS [
> 0.504 s]
> [INFO] nifi-kafka-1-0-nar . SUCCESS [
> 0.537 s]
> [INFO] nifi-kite-bundle ... SUCCESS [
> 0.014 s]
> [INFO] nifi-kite-processors ... SUCCESS [
> 13.303 s]
> [INFO] nifi-kite-nar .. SUCCESS [
> 1.225 s]
> [INFO] nifi-hadoop-record-utils ... SUCCESS [
> 1.459 s]
> [INFO] nifi-kudu-bundle ... SUCCESS [
> 0.013 s]
> [INFO] nifi-kudu-processors ... SUCCESS [
> 5.861 s]
> [INFO] nifi-kudu-nar .. SUCCESS [
> 0.897 s]
> [INFO] nifi-solr-bundle ... SUCCESS [
> 0.018 s]
> [INFO] nifi-solr-processors ... SUCCESS [
> 17.024 s]
> [INFO] nifi-solr-nar .. SUCCESS [
> 0.742 s]
> [INFO] nifi-confluent-platform-bundle . SUCCESS [
> 0.018 s]
> [INFO] nifi-confluent-schema-registry-service . SUCCESS [
> 1.693 s]
> [INFO] nifi-confluent-platform-nar  SUCCESS [
> 0.742 s]
> [INFO] nifi-aws-bundle  SUCCESS [
> 0.014 s]
> [INFO] nifi-aws-service-api ... SUCCESS [
> 1.236 s]
> [INFO] nifi-aws-abstract-processors ... SUCCESS [
> 2.407 s]
> [INFO] nifi-aws-processors  SUCCESS [
> 17.781 s]
> [INFO] nifi-aws-service-api-nar ... SUCCESS [
> 0.360 s]
> [INFO] nifi-aws-nar ... SUCCESS [
> 3.306 s]
> [INFO] nifi-social-media-bundle ... SUCCESS [
> 0.017 s]
> [INFO] nifi-twitter-processors  SUCCESS [
> 4.267 s]
> [INFO] nifi-social-media-nar .. SUCCESS [
> 0.266 s]
> [INFO] nifi-enrich-bundle . SUCCESS [
> 0.014 s]
> [INFO] nifi-enrich-processors . SUCCESS [
> 15.033 s]
> [INFO] nifi-enrich-nar  SUCCESS [
> 0.442 s]
> [INFO] nifi-hl7-bundle  SUCCESS [
> 0.020 s]
> [INFO] nifi-hl7-processors  SUCCESS [
> 5.830 s]
> [INFO] nifi-hl7-nar ... SUCCESS [
> 1.049 s]
> [INFO] nifi-ccda-bundle ... SUCCESS [
> 0.037 s]
> [INFO] nifi-ccda-processors ... SUCCESS [02:14
> min]
> [INFO] nifi-ccda-nar .. SUCCESS [
> 0.795 s]
> [INFO] nifi-language-translation-bundle ... SUCCESS [
> 0.013 s]
> [INFO] nifi-yandex-processors . SUCCESS [
> 4.358 s]
> [INFO] nifi-language-translation-nar .. SUCCESS [
> 0.501 s]
> [INFO] nifi-mongodb-bundle ...

Re: Empty String in UpdateAttribute

[Brandon DeVries]

Joe,

Primarily, I believe the use case is legacy flows pre-dating the existence
of "delete attribute" functionality. The most "correct" solution may be to
delete the attribute, but that doesn't mean we should break backwards
compatibility.

Brandon
On Fri, Jan 19, 2018 at 5:33 PM Joe Percivall  wrote:

> Hey Mark,
>
> Yeah, at the time I couldn't think of a use-case that a user would want to
> set an attribute to empty so when I added the validator for the stateful
> portion, I added the empty validator as well. That said, I don't see a need
> to keep it if it's preventing your use-case. Curious, what is your use-case
> that you always need to set an attribute to an empty string?
>
> - Joe
>
>
>
> On Fri, Jan 19, 2018 at 2:43 PM, Michael Moser  wrote:
>
> > Hi Mark,
> >
> > After reading the JIRA [1] and the PR for the UpdateAttribute
> > modifications, I believe this was simply an unintended consequence of the
> > code changes.  If you would like to write a new JIRA, then I'm sure we
> can
> > restore the "Set Empty String" functionality when not storing attributes
> in
> > local state.
> >
> > [1] - https://issues.apache.org/jira/browse/NIFI-1582
> >
> > -- Mike
> >
> >
> > On Thu, Jan 18, 2018 at 5:11 PM, Mark Bean 
> wrote:
> >
> > > UpdateAttribute supports stateful properties. In doing so, the
> validator
> > > applied to the attributes changes. In the case where stateful
> properties
> > > are not desired, the validator adds StandardValidators.NON_EMPTY_
> > > VALIDATOR.
> > > This means the "Set Empty String" checkbox cannot be used.
> > >
> > > One solution is to store state locally, and then set both the Stateful
> > > Variables Initial Value and the dynamic property to the Empty String.
> > >
> > > However, my question: why is the NON_EMPTY_VALIDATOR added (when
> stateful
> > > is not set) versus just the ATTRIBUTE_KEY_PROPERTY_NAME_VALIDATOR?
> > >
> > > Thanks,
> > > Mark
> > >
> >
>
>
>
> --
> *Joe Percivall*
> linkedin.com/in/Percivall
> e: jperciv...@apache.com
>

Re: Web site fixes/corrections

[Brandon DeVries]

I think Benson actually asked off of the  PMC[1], so he should probably
just be removed from the list.

(Obviously make sure I read that right.  I have no desire to kick him out
if he doesn't actually want out...)

[1]
http://mail-archives.apache.org/mod_mbox/incubator-general/201507.mbox/%3CCALhtWkd6%2BwH%3DyAhOkD-X-7NC8J_7A-FmU8wzYgK%2BTiXG%2BwjF7A%40mail.gmail.com%3E

On Wed, Jul 22, 2015 at 7:52 AM Dan Bress  wrote:

> If you are still working on this Aldrin, I noticed that Benson's name is
> spelled wrong on the people page [1]
> bimargulies Benson Margulie
>
> [1]http://nifi.apache.org/people.html
>
> Dan Bress
> Software Engineer
> ONYX Consulting Services
>
> 
> From: Joe Witt 
> Sent: Tuesday, July 21, 2015 6:02 PM
> To: dev@nifi.apache.org
> Subject: Re: Web site fixes/corrections
>
> I think Sean's proposal is a good balance.  We want it to be easy for
> folks to contribute to making the website awesome-er.  We have a good
> start but we have a long way to go.  For simple cleanup/maintenance
> things a single cover-all JIRA should suffice.  If someone wants to
> work a new concept/page/theme then that should have its own ticket.
>
> On Tue, Jul 21, 2015 at 3:00 PM, Sean Busbey  wrote:
> > I think a single "cleanup" jira that covers everything fixed in one go
> > would be good, but I know that I tend to fall on the jira-for-everything
> > side of things.
> >
> > On Tue, Jul 21, 2015 at 4:56 PM, Aldrin Piri 
> wrote:
> >
> >> All,
> >>
> >> I have made fixes to the website inclusive of items such as correcting
> >> broken links, typos, and the like and have never created issues for
> these
> >> fixes as I viewed them to be minor changes.
> >>
> >> I was curious if this practice seemed acceptable or if the community
> >> preferred an issue for each such change.
> >>
> >> Thanks!
> >>
> >
> >
> >
> > --
> > Sean
>

Re: Help required with my custom controller service

[Brandon DeVries]

All,

Was this ever solved / explained?  I'm having a similar issue.  If there's
a simple answer that I'm missing that would be great... otherwise I might
post an example set of NARs somewhere tomorrow.  Probably the simplest
question first is, if I want to create a custom processor that uses a
SSLContextService, what dependencies do I need to add where (assuming we're
using the processor NAR archetype)?  I've created an example that compiles,
loads in NiFi, and can be dropped on the graph.  However, it can not be
configured to use any currently existing SSLContextService, and any that
are created have the same UUID issue described by David.  Thoughts?

Brandon



On Tue, Jul 28, 2015 at 10:50 AM Mark Payne  wrote:

> Dave,
>
> In addition to the points that Aldrin brought up, are your CacheService
> interface
> and StandardCacheService classes in the same NAR?
>
> If they are not in the same NAR, does your NAR for StandardCacheService
> have a Maven dependency
> on the NAR that houses CacheService?
>
> Thanks
> -Mark
>
> 
> > From: aldrinp...@gmail.com
> > Date: Mon, 27 Jul 2015 23:26:58 -0400
> > Subject: Re: Help required with my custom controller service
> > To: dev@nifi.apache.org; davidrsm...@btinternet.com
> >
> > Dave,
> >
> > Some quick notes as I work through your setup.
> >
> > Did you create an entry in your
> > NAR's org.apache.nifi.controller.ControllerService file? This would be in
> > src/main/resources/META-INF/services directory of your project. It seems
> > like this isn't likely the case, but a point that can sometimes get
> > overlooked.
> >
> > Your accessing of the service property in your processor looks good.
> >
> > Did you accidentally override AbstractControllerService#getIdentifier()?
> > It seems like this could lead to the error you are seeing, but it is hard
> > to be sure.
> >
> > If you are able to share more code with us, we can provide some more
> > pointed assistance. Have you managed to get your processor and controller
> > service running within the test framework [1]? If not, I would suggest
> > doing so to cut down on iterations in debugging your problems as well as
> > providing a means for you to easily step through the code.
> >
> > [1]
> http://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#testing
> >
> > On Sun, Jul 26, 2015 at 4:45 PM, DAVID SMITH  >
> > wrote:
> >
> >> Hi
> >> I have been trying to write a simple Cache Controller Service that uses
> a
> >> Map to hold  key/value pairs. I am using
> >> nifi-0.1.0-incubating. I have created an Interface called CacheService
> >> which extends ControllerService.public interface CacheService extends
> >> ControllerService {
> >>
> >> I have created an implementation called StandardCacheService which is as
> >> follows:
> >> public class StandardCacheService extends AbstractControllerService
> >> implements CacheService {
> >> I have compiled the StandardCacheService and created a nar which I have
> >> then put into my NiFi lib directory, my StandardCacheService Controller
> >> Service appears in the Controller services tab of the NiFi Flow Settings
> >> panel, I can edit it and enable it. Also there are no abnormal log
> messages.
> >> However, when I try to configure the CacheService in my test Processor,
> >> again I get no errors but when I cannot see my StandardCacheService. My
> >> processor has the following snippets to implement the Cache
> >> Service/Controller:
> >> public static final PropertyDescriptor CACHE_SERVICE = new
> >> PropertyDescriptor.Builder()
> >> .name("Cache Service")
> >> .description("The Controller Service to use in order to obtain a
> >> Cache Service")
> >> .required(false)
> >> .identifiesControllerService(CacheService.class)
> >> .build();
> >>
> >> final CacheService cache =
> >>
> context.getProperty(CACHE_SERVICE).asControllerService(CacheService.class);
> >>
> >>
> >>
> >>
> >> When configuring the processor, the 'Cache Service' drop down doesn't
> show
> >> a populated list showing my StandardCacheService, but it does allow me
> to
> >> create a new service, the resulting panel allows me to then select
> >> StandardCacheService. When I select 'create' I get a UUID value in the
> >> Value property of the Configure Processor panel. When I then apply this
> >> the processor shows a yellow triangle, when I hover over it I get the
> >> following message:
> >> 'Cache Service validated against  is invalid because Invalid
> >> Controller service  is not a valid Controller service Identifier
> or
> >> does not reference the correct type of Controller service.'
> >> Can you help please
> >> Many thanksDave
> >>
> >>
>

Re: Help required with my custom controller service

[Brandon DeVries]

Bryan and Mark,

I suspect that will do it.  I'll check tomorrow.

Brandon

On Mon, Aug 10, 2015, 4:23 PM Bryan Bende  wrote:

> Not sure if this is helpful, but there is a section about adding
> dependencies on controller services at the end of this wiki page:
>
>
> https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Custom+Processors
>
>
> On Monday, August 10, 2015, Mark Payne  wrote:
>
> > Brandon,
> >
> > I've seen this occur when I forgot to add a dependency on the
> > ssl-context-service-api-nar in my NAR's pom.
> >
> > You should have something like:
> > 
> >   org.apache.nifi
> >   ssl-context-service-api
> >   0.3.0-SNAPSHOT
> >   nar
> > 
> >
> > Is that by chance the issue, or is there something else going on?
> >
> > Thanks
> > -Mark
> >
> > 
> > > From: b...@jhu.edu 
> > > Date: Mon, 10 Aug 2015 19:13:50 +
> > > Subject: Re: Help required with my custom controller service
> > > To: dev@nifi.apache.org 
> > >
> > > All,
> > >
> > > Was this ever solved / explained? I'm having a similar issue. If
> there's
> > > a simple answer that I'm missing that would be great... otherwise I
> might
> > > post an example set of NARs somewhere tomorrow. Probably the simplest
> > > question first is, if I want to create a custom processor that uses a
> > > SSLContextService, what dependencies do I need to add where (assuming
> > we're
> > > using the processor NAR archetype)? I've created an example that
> > compiles,
> > > loads in NiFi, and can be dropped on the graph. However, it can not be
> > > configured to use any currently existing SSLContextService, and any
> that
> > > are created have the same UUID issue described by David. Thoughts?
> > >
> > > Brandon
> > >
> > >
> > >
> > > On Tue, Jul 28, 2015 at 10:50 AM Mark Payne  > > wrote:
> > >
> > >> Dave,
> > >>
> > >> In addition to the points that Aldrin brought up, are your
> CacheService
> > >> interface
> > >> and StandardCacheService classes in the same NAR?
> > >>
> > >> If they are not in the same NAR, does your NAR for
> StandardCacheService
> > >> have a Maven dependency
> > >> on the NAR that houses CacheService?
> > >>
> > >> Thanks
> > >> -Mark
> > >>
> > >> 
> > >>> From: aldrinp...@gmail.com 
> > >>> Date: Mon, 27 Jul 2015 23:26:58 -0400
> > >>> Subject: Re: Help required with my custom controller service
> > >>> To: dev@nifi.apache.org ; davidrsm...@btinternet.com
> > 
> > >>>
> > >>> Dave,
> > >>>
> > >>> Some quick notes as I work through your setup.
> > >>>
> > >>> Did you create an entry in your
> > >>> NAR's org.apache.nifi.controller.ControllerService file? This would
> be
> > in
> > >>> src/main/resources/META-INF/services directory of your project. It
> > seems
> > >>> like this isn't likely the case, but a point that can sometimes get
> > >>> overlooked.
> > >>>
> > >>> Your accessing of the service property in your processor looks good.
> > >>>
> > >>> Did you accidentally override
> > AbstractControllerService#getIdentifier()?
> > >>> It seems like this could lead to the error you are seeing, but it is
> > hard
> > >>> to be sure.
> > >>>
> > >>> If you are able to share more code with us, we can provide some more
> > >>> pointed assistance. Have you managed to get your processor and
> > controller
> > >>> service running within the test framework [1]? If not, I would
> suggest
> > >>> doing so to cut down on iterations in debugging your problems as well
> > as
> > >>> providing a means for you to easily step through the code.
> > >>>
> > >>> [1]
> > >>
> http://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#testing
> > >>>
> > >>> On Sun, Jul 26, 2015 at 4:45 PM, DAVID SMITH <
> > davidrsm...@btinternet.com 
> > >>>
> > >>> wrote:
> > >>>
> >  Hi
> >  I have been trying to write a simple Cache Controller Service that
> > uses
> > >> a
> >  Map to hold  key/value pairs. I am using
> >  nifi-0.1.0-incubating. I have created an Interface called
> CacheService
> >  which extends ControllerService.public interface CacheService
> extends
> >  ControllerService {
> > 
> >  I have created an implementation called StandardCacheService which
> is
> > as
> >  follows:
> >  public class StandardCacheService extends AbstractControllerService
> >  implements CacheService {
> >  I have compiled the StandardCacheService and created a nar which I
> > have
> >  then put into my NiFi lib directory, my StandardCacheService
> > Controller
> >  Service appears in the Controller services tab of the NiFi Flow
> > Settings
> >  panel, I can edit it and enable it. Also there are no abnormal log
> > >> messages.
> >  However, when I try to configure the CacheService in my test
> > Processor,
> >  again I get no errors but when I cannot see my StandardCacheService.
> > My
> >  processor has the following snippets to implement the Cache
> >  Service/Controller:
> >  p

Re: [DISCUSS] Feature proposal: Read-only mode as default

[Brandon DeVries]

I think "undo" is the most important part here.  I agree with Alex's
statement, "accidents happen; I prefer to enable users to learn from
mistakes and exercise more care next time."  Delete confirmations may be
fine (although I suspect they would begin to annoy me fairly quickly... if
it's just for things not visible on the graph its probably fine), and
reducing accidental movement is all well and good.  But ultimately, if i
can say "whoops, undo" then they don't really matter.  And as a side
effect, maybe I'll be more careful next time.  Also, who says I'll remember
that I left the graph in "edit" mode?  If I get pulled to something else
and come back, and have been trained that "read only" is default, then I
run the risk of being bitten by the same things "edit" vs. "read only" was
supposed to prevent.  I would suggest focusing on implementing "undo",
getting that in people's hands, and then seeing if after having done that
anyone still cares about the other issues .

Brandon

On Tue, Aug 11, 2015 at 8:28 AM, Joe Witt  wrote:

> To summarize where we're at ...
>
> Proposed approaches (summary):
>
> - Establish a default read-only view whereby an operator can enable
> edit mode.  Use confirmation dialogs for deletes.
>
> - Keep the current model but add support for ‘undo’.
>
> - Let the user choose whether to lock their view or not as user preference.
>
> - For delete add more protections to make accidents less likely and
> for movement provide an explicit ‘move action’.
>
> The idea of locking seems to have some strong views on both sides and
> both sides have even been argued by the same people (i now count
> myself among that group).
>
> It looks like a consensus view there though is:
>
> - Try to make panning the view of the flow and moving components on
> the flow two specific/discrete actions to avoid accidental movement.
>
> - Add support for undo
>
> - Provide sufficient dialog/protection for delete cases.
>
> This preserves the model whereby we generally trust that the user will
> do the right thing and we’ll do more to help them and that when they
> don’t they will learn and have help to promptly restore a good state.
> How do folks feel about that?
>
> Thanks
> Joe
>
> On Tue, Aug 11, 2015 at 5:11 AM, Alex Moundalexis 
> wrote:
> > Counterpoint, accidents happen; I prefer to enable users to learn from
> > mistakes and exercise more care next time. You can't remove every mildly
> > sharp edge without impacting experienced operators; resist the urge at a
> few
> > comments to dumb down the tool.
> >
> > If some protection is added to the UI, suggest an option for "expert
> mode"
> > that retains original functionality... that way experienced operators
> aren't
> > affected.
> >
> > Alex Moundalexis
> > Senior Solutions Architect
> > Cloudera Partner Engineering
> >
> > Sent from a mobile device; please excuse brevity.
> >
> > On Aug 7, 2015 10:31 PM, "Joe Witt"  wrote:
> >>
> >> Team,
> >>
> >> We've been hearing from users of nifi that it is too easy to
> >> accidentally move something on the flow or delete large portions of
> >> the flow.  This is the case when panning the screen for example and
> >> accidentally having a processor selected instead.
> >>
> >> What is worth consideration then is the notion of making the flow
> >> 'read-only' by default.  In this case the user would need to
> >> explicitly 'enable edit mode'.  We would then also support a
> >> confirmation dialog or similar construct whenever deleting components
> >> on the flow.
> >>
> >> Anyone have a strong objection to this concept?  If so, do you have an
> >> alternative in mind that would help avoid accidental movement?
> >>
> >> Thanks
> >> Joe
>

Re: I'd like to OPT out of the email lists

[Brandon DeVries]

James,

Instructions on managing your Apache mailing list subscriptions can be
found here:

http://apache.org/foundation/mailinglists.html

Brandon

On Tue, Aug 11, 2015 at 12:14 PM James Cornell 
wrote:

> I'd like to OPT out of the email lists.
>

Re: Help required with my custom controller service

[Brandon DeVries]

All,

Looks like I should be good now.  Thanks.

Brandon

On Mon, Aug 10, 2015 at 4:39 PM DAVID SMITH 
wrote:

> All
>
> That certainly cured the problem in my case.
>
> Dave
>
> Sent from Yahoo! Mail on Android
>
> ------
> * From: * Brandon DeVries ;
> * To: * dev@nifi.apache.org ;
> * Subject: * Re: Help required with my custom controller service
> * Sent: * Mon, Aug 10, 2015 8:25:58 PM
>
> Bryan and Mark,
>
> I suspect that will do it.  I'll check tomorrow.
>
> Brandon
>
> On Mon, Aug 10, 2015, 4:23 PM Bryan Bende  wrote:
>
> > Not sure if this is helpful, but there is a section about adding
> > dependencies on controller services at the end of this wiki page:
> >
> >
> >
> https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Custom+Processors
> >
> >
> > On Monday, August 10, 2015, Mark Payne  wrote:
> >
> > > Brandon,
> > >
> > > I've seen this occur when I forgot to add a dependency on the
> > > ssl-context-service-api-nar in my NAR's pom.
> > >
> > > You should have something like:
> > > 
> > >  org.apache.nifi
> > >  ssl-context-service-api
> > >  0.3.0-SNAPSHOT
> > >  nar
> > > 
> > >
> > > Is that by chance the issue, or is there something else going on?
> > >
> > > Thanks
> > > -Mark
> > >
> > > 
> > > > From: b...@jhu.edu 
> > > > Date: Mon, 10 Aug 2015 19:13:50 +
> > > > Subject: Re: Help required with my custom controller service
> > > > To: dev@nifi.apache.org 
> > > >
> > > > All,
> > > >
> > > > Was this ever solved / explained? I'm having a similar issue. If
> > there's
> > > > a simple answer that I'm missing that would be great... otherwise I
> > might
> > > > post an example set of NARs somewhere tomorrow. Probably the simplest
> > > > question first is, if I want to create a custom processor that uses a
> > > > SSLContextService, what dependencies do I need to add where (assuming
> > > we're
> > > > using the processor NAR archetype)? I've created an example that
> > > compiles,
> > > > loads in NiFi, and can be dropped on the graph. However, it can not
> be
> > > > configured to use any currently existing SSLContextService, and any
> > that
> > > > are created have the same UUID issue described by David. Thoughts?
> > > >
> > > > Brandon
> > > >
> > > >
> > > >
> > > > On Tue, Jul 28, 2015 at 10:50 AM Mark Payne  > > > wrote:
> > > >
> > > >> Dave,
> > > >>
> > > >> In addition to the points that Aldrin brought up, are your
> > CacheService
> > > >> interface
> > > >> and StandardCacheService classes in the same NAR?
> > > >>
> > > >> If they are not in the same NAR, does your NAR for
> > StandardCacheService
> > > >> have a Maven dependency
> > > >> on the NAR that houses CacheService?
> > > >>
> > > >> Thanks
> > > >> -Mark
> > > >>
> > > >> 
> > > >>> From: aldrinp...@gmail.com 
> > > >>> Date: Mon, 27 Jul 2015 23:26:58 -0400
> > > >>> Subject: Re: Help required with my custom controller service
> > > >>> To: dev@nifi.apache.org ; davidrsm...@btinternet.com
> > > 
> > > >>>
> > > >>> Dave,
> > > >>>
> > > >>> Some quick notes as I work through your setup.
> > > >>>
> > > >>> Did you create an entry in your
> > > >>> NAR's org.apache.nifi.controller.ControllerService file? This would
> > be
> > > in
> > > >>> src/main/resources/META-INF/services directory of your project. It
> > > seems
> > > >>> like this isn't likely the case, but a point that can sometimes get
> > > >>> overlooked.
> > > >>>
> > > >>> Your accessing of the service property in your processor looks
> good.
> > > >>>
> > > >>> Did you accidentally override
> > > AbstractControllerService#getIdentifier()?
> > > >>> It seems like th

Re: [DISCUSS] Removal of the 'master' vs 'develop' distinction

[Brandon DeVries]

Personally, I still think GitLab Flow[1] is all we need for us to be Really
Useful Engines.

[1] https://about.gitlab.com/2014/09/29/gitlab-flow/

Brandon

On Thu, Aug 13, 2015 at 2:15 PM Joe Witt  wrote:

> Resending
> On Aug 13, 2015 12:22 PM, "Joe Witt"  wrote:
>
> > Team,
> >
> > It was proposed by Ryan Blue on another thread that we consider
> > dropping the master vs develop distinction.  In the interest of his,
> > in my view, very good point I didn't want it to get buried in that
> > thread.
> >
> > [1] is the thread when we last discussed gitflow/develop/master on
> > entry to the incubator.
> >
> > And from that thread here is the part I wish I had better understood
> > when the wise Mr Benson said it:
> >
> > "Another issue with gitflow is the master branch. The master branch is
> > supposed to get merged to for releases. The maven-release-plugin won't
> > do that, and the jgitflow plugin is unsafe. So one option is to 'use
> > gitflow' but not bother with the master versus develop distinction,
> > the other is to do manual merges to master at release points."
> >
> > I think we should follow this guidance: "'use gitflow' but not bother
> > with the master versus develop distinction".  I say this from having
> > done the release management job now a couple of times including having
> > done a 'hotfix'.
> >
> > My comments here are not a rejection of that master/develop concept in
> > general.  It is simply pointing out that for the Apache NiFi community
> > it is not adding value but is creating confusion and delay [2].
> >
> > Thanks
> > Joe
> >
> > [1] http://s.apache.org/GIW
> > [2] Sir Topham Hatt - Thomas and Friends (tm)
> >
>

0.3.0 release

[Brandon DeVries]

All,

It looks like there's only one ticket (NIFI-936[1]) unresolved that is
slated for 0.3.0[2].  Are there any other issues that anyone believes need
to be in the 0.3.0 release?  If not, does anyone object to shooting for an
0.3.0 release shortly after NIFI-936 is resolved?

[1] https://issues.apache.org/jira/browse/NIFI-936
[2]
https://issues.apache.org/jira/issues/?filter=12331975&jql=project%20%3D%20NIFI%20AND%20fixVersion%20%3D%200.3.0%20%20order%20by%20status%20asc

Brandon

Re: 0.3.0 release

[Brandon DeVries]

Brian,

In looking at the ticket, it sounds like you're almost satisfied with the
fix.  Can you let us know when you're convinced it's working?  Thanks.

Brandon

On Tue, Sep 8, 2015 at 3:51 PM Brian Ghigiarelli 
wrote:

> Hi Brandon,
>
> With the latest 0.3.0-SNAPSHOT on master, I've started seeing an issue
> appending content to FlowFiles (not reproducible with unit tests, but is
> reproducible by running the GetKafka processor when it is configured for
> batching).
>
> Because a big part of the Content Repo was reworked with this release, I'd
> ask that you consider a ticket I just created:
> https://issues.apache.org/jira/browse/NIFI-938
>
> Thanks,
> Brian
>
> On Tue, Sep 8, 2015 at 3:05 PM, Brandon DeVries  wrote:
>
> > All,
> >
> > It looks like there's only one ticket (NIFI-936[1]) unresolved that is
> > slated for 0.3.0[2].  Are there any other issues that anyone believes
> need
> > to be in the 0.3.0 release?  If not, does anyone object to shooting for
> an
> > 0.3.0 release shortly after NIFI-936 is resolved?
> >
> > [1] https://issues.apache.org/jira/browse/NIFI-936
> > [2]
> >
> >
> https://issues.apache.org/jira/issues/?filter=12331975&jql=project%20%3D%20NIFI%20AND%20fixVersion%20%3D%200.3.0%20%20order%20by%20status%20asc
> >
> > Brandon
>

Re: [VOTE] Release Apache NiFi 0.3.0

[Brandon DeVries]

To copy and paste from Mark:

Downloaded source, verified checksums and that the key/signature was valid.

Was able to build with contrib-check without any problems.

README/LICENSE/NOTICE all look good. Application runs without any problems.

+1 (binding) - Release this package as nifi-0.3.0

Brandon

On Tue, Sep 15, 2015 at 12:18 PM Joe Percivall
 wrote:

> Whops that +1 should have been not binding.
>
> - - - - - -
> Joseph Percivall
> linkedin.com/in/Percivall
> e: joeperciv...@yahoo.com
>
>
>
>
> On Tuesday, September 15, 2015 12:08 PM, Joe Percivall <
> joeperciv...@yahoo.com.INVALID> wrote:
>
>
>
> Checked the keys, signature and verified the checksums.Valid
> README/LICENSE/NOTICE.Built with mvn clean install -Pcontrib-check.Did a
> test dataflow and template, worked as epected.
> [+1] (binding)
> - - - - - - Joseph Percivalllinkedin.com/in/Percivalle:
> joeperciv...@yahoo.com
>
>
>
>
>  On Tuesday, September 15, 2015 10:45 AM, Bryan Bende <
> bbe...@gmail.com> wrote:
>
>
> Signature and checksums look good
> Build passes with contrib-check
> README/LICENSE/NOTICE all look good
> Test dataflows perform as expexted
>
> +1 (binding) - Release this package as nifi-0.3.0
>
>
> On Tue, Sep 15, 2015 at 9:50 AM, Mark Payne  wrote:
>
> > Downloaded source, verified checksums and that the key/signature was
> valid.
> >
> > Was able to build with contrib-check without any problems.
> >
> > README/LICENSE/NOTICE all look good. Application runs without any
> problems.
> >
> > +1 (binding) - Release this package as nifi-0.3.0
> >
> > Thanks
> > -Mark
> >
> >
> >
> > > On Sep 14, 2015, at 11:13 PM, Matt Gilman 
> > wrote:
> > >
> > > Hello
> > > I am pleased to be calling this vote for the source release of Apache
> > NiFi
> > > 0.3.0.
> > >
> > > The source zip, including signatures, digests, etc. can be found at:
> > > https://repository.apache.org/content/repositories/orgapachenifi-1060
> > >
> > > The Git tag is nifi-0.3.0-RC1
> > > The Git commit ID is 2ec735e35025fed3c63d51128ec0609ffe1fa7e3
> > >
> >
> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=2ec735e35025fed3c63d51128ec0609ffe1fa7e3
> > >
> > > Checksums of nifi-0.3.0-source-release.zip:
> > > MD5: 0bca350d5d6d9c9a459304253b8121c4
> > > SHA1: 4b14bf1c0ddc3d970ef44dac93e716e9e6964842
> > >
> > > Release artifacts are signed with the following key:
> > > https://people.apache.org/keys/committer/mcgilman.asc
> > >
> > > KEYS file available here:
> > > https://dist.apache.org/repos/dist/release/nifi/KEYS
> > >
> > > 89 issue was closed/resolved for this release:
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12329653
> > >
> > > Release note highlights can be found here:
> > >
> >
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version0.3.0
> > >
> > > The vote will be open for 72 hours.
> > > Please download the release candidate and evaluate the necessary items
> > > including checking hashes, signatures, build from source, and test.
> The
> > > please vote:
> > >
> > > [ ] +1 Release this package as nifi-0.3.0
> > > [ ] +0 no opinion
> > > [ ] -1 Do not release this package because...
> >
> >
>

Re: MonitorActivity bug?

[Brandon DeVries]

Sumanth,

The REL_ACTIVITY_RESTORED relationship is triggered by an incoming FlowFile
after a period of inactivity. Those are the attributes that are copied. The
REL_INACTIVE relationship is triggered by the absence of a FlowFile for
some period, so there are no attributes to copy. Another implementation may
be possible, but that is why it currently behaves as it does.

Brandon
On Sun, Oct 18, 2015 at 1:46 AM Sumanth Chinthagunta 
wrote:

> Noticed "copy attributes" is implemented only for  REL_ACTIVITY_RESTORED
> case.
> I needed original attributes for REL_INACTIVE case as well.
> wonder why it is limited to activity restored relationship only.
>
>
>
> https://github.com/apache/nifi/blob/31fba6b3332978ca2f6a1d693f6053d719fb9daa/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/MonitorActivity.java#L208
>
> --
> Thanks,
> Sumanth
>

Re: nifi fails build on CentOS 7

[Brandon DeVries]

Jeff,

Can you see what tests had failures?  Also, if you just want to verify that
your system can do the build, you could do something like "mvn clean
install -DskipTests".  Let us know what you see.  Thanks.

Brandon

On Wed, Nov 4, 2015 at 8:46 AM, Jeff  wrote:

>
> Well, the build got much further with the Oracle JDK but still failed;
>
> [INFO]
> 
> [INFO] BUILD FAILURE
> [INFO]
> 
> [INFO] Total time: 07:47 min (Wall Clock)
> [INFO] Finished at: 2015-11-03T17:07:40-06:00
> [INFO] Final Memory: 183M/564M
> [INFO]
> 
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:2.18:test (default-test) on
> project nifi-framework-core: There are test failures.
> [ERROR]
> [ERROR] Please refer to
> /root/nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/target/surefire-reports
> for the individual test results.
>
>
>
> > On Nov 3, 2015, at 6:54 PM, Cerulean Blue  wrote:
> >
> > Oh, I also adjusted the memory!
> >
> > Sent from my iPhone
> >
> >> On Nov 3, 2015, at 6:50 PM, Joe Witt  wrote:
> >>
> >> Jeff - can you please send another note telling Tony that it turns out
> >> it was memory settings :-)
> >>
> >> He is making fun of me off-list.  He should totally do that on-list!
> >>
> >> Glad you're in business now and sorry for my terrible misdirection.
> >>
> >> Joe
> >>
> >>> On Wed, Nov 4, 2015 at 12:49 AM, Tony Kurc  wrote:
> >>> Huzzah
> >>>
>  On Tue, Nov 3, 2015 at 7:48 PM, Jeff  wrote:
> 
> 
>  I removed openjdk and installed the Oracle jdk and it now works.
> 
>  Thanks for the help!!
> 
> > On Nov 3, 2015, at 6:40 PM, Tony Kurc  wrote:
> >
> > it is entirely possible that you only have the jre installed. 'yum
> > whatprovides "*/bin/javac"' may tell you packages which you can
> install
> > which include a compiler.
> >
> >> On Tue, Nov 3, 2015 at 7:35 PM, Jeff  wrote:
> >>
> >>
> >>
> >> find / -name javac did not return anything
> >>
> >> I had JAVA_HOME set to
> >> /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.65-2.b17.el7_1.x86_64/jre
> >>
> >>
> >>
> >>> On Nov 3, 2015, at 6:30 PM, Tony Kurc  wrote:
> >>>
> >>> Jeff,
> >>> Were you able to run that find command?
> >>>
>  On Tue, Nov 3, 2015 at 7:09 PM, Jeff  wrote:
> 
> 
>  [INFO] BUILD FAILURE
>  [INFO]
> 
> 
>  [INFO] Total time: 8.091 s
>  [INFO] Finished at: 2015-11-03T12:08:59-06:00
>  [INFO] Final Memory: 76M/365M
>  [INFO]
> 
> 
>  [ERROR] Failed to execute goal
>  org.apache.maven.plugins:maven-compiler-plugin:3.2:compile
>  (default-compile) on project nifi-api: Compilation failure ->
> [Help 1]
>  [ERROR]
>  [ERROR] To see the full stack trace of the errors, re-run Maven
> with
>  the
>  -e switch.
>  [ERROR] Re-run Maven using the -X switch to enable full debug
> logging.
>  [ERROR]
>  [ERROR] For more information about the errors and possible
> solutions,
>  please read the following articles:
>  [ERROR] [Help 1]
> 
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
>  [ERROR]
>  [ERROR] After correcting the problems, you can resume the build
> with
>  the
>  command
>  [ERROR]   mvn  -rf :nifi-api
> 
> 
> 
> > On Nov 3, 2015, at 6:07 PM, Joe Witt  wrote:
> >
> > Can you go ahead and try just plain ol'
> >
> > mvn clean install
> >
> > I am curious if there is a memory/settings issue that is
> manifesting
> > as an exception during compilation but that it isn't actually a
> > compilation problem at all...
> >
> >> On Wed, Nov 4, 2015 at 12:06 AM, Jeff 
> wrote:
> >> The only other message I see is
> >>
> >> [WARNING] Unable to autodetect 'javac' path, using 'javac' from
> the
>  environment.
> >>
> >> No worries, it can wait.
> >>
> >> Thanks,
> >>
> >>> On Nov 3, 2015, at 5:46 PM, Joe Witt 
> wrote:
> >>>
> >>> if i weren't on horrible hotel wifi right now i'd try to pull
> down
> >>> centos.  Is there anymore that can be found with those stack
>  traces .
> >>> They're not saying much.  Your java and maven versions seem
> quite
> >> good
> >>> so need to dig further.
> >>>
>  On Tue, Nov 3, 2015 at 11:42 PM, Jeff 
> wrote:
> >

Re: ExecuteStreamCommand tests

[Brandon DeVries]

I would undo the removal for now, and make a point of doing the test
properly. I don't like the idea of removing the test and saying we'll add
new ones eventually (those sorts of things tend to not happen...).

Brandon
On Thu, Nov 12, 2015 at 5:36 PM Tony Kurc  wrote:

> Shipping built jars that tests depend on is icky. Not shipping the source
> to those tests is ickier.
> On Nov 12, 2015 5:34 PM, "Joe Witt"  wrote:
>
> > i think we should kill those tests which depend on the build of those
> > jars personally.  But if the view is to undo the removal of those
> > three classes i can do that.
> >
> > Thanks
> > Joe
> >
> > On Thu, Nov 12, 2015 at 5:32 PM, Tony Kurc  wrote:
> > > Do you plan to undo the removal?
> > > On Nov 12, 2015 4:46 PM, "Joe Witt"  wrote:
> > >
> > >> well that explains these goofball classes I deleted the other day
> > >>
> > >> https://issues.apache.org/jira/browse/NIFI-1134
> > >>
> > >> These classes were used to make those Jars.  Those jars are used to
> > >> test execute command.  We've now removed the source that was floating
> > >> randomly.  We need the built to automatically create whatever we
> > >> execute against if we're going to do this.  Those tests should be
> > >> replaced by something else.
> > >>
> > >>
> > >> On Thu, Nov 12, 2015 at 3:02 PM, Joe Percivall
> > >>  wrote:
> > >> > Tony,
> > >> >
> > >> > I did a bit of digging through the history and the jars were a part
> of
> > >> the initial code import so unless if Joe or someone else knows where
> > they
> > >> came from then we may be out of luck.
> > >> >
> > >> > Joe
> > >> >
> > >> > - - - - - -
> > >> > Joseph Percivall
> > >> > linkedin.com/in/Percivall
> > >> > e: joeperciv...@yahoo.com
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > On Wednesday, November 11, 2015 6:23 PM, Tony Kurc <
> trk...@gmail.com>
> > >> wrote:
> > >> >
> > >> >
> > >> >
> > >> > All, I was code reviewing and something occurred to me. This raised
> my
> > >> > eyebrow:
> > >> >
> > >>
> >
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestExecuteStreamCommand.java#L63
> > >> >
> > >> > If I'm reading it right, the test "runs" a jar that we've got in our
> > >> source
> > >> > tree
> > >> >
> > >> > What code made those jars in src/test/resources?
> > >>
> >
>

Re: Questions about the ordering of the FlowFile.

[Brandon DeVries]

Paresh,

 You might want to look at the PriorityAttributePrioritizer[1]:

*PriorityAttributePrioritizer*: Given two FlowFiles that both have a
"priority" attribute, the one that has the highest priority value will be
prprocessed first. Note that an UpdateAttribute processor should be used to
add the "priority" attribute to the FlowFiles before they reach a
connection that has this prioritizer set. Values for the "priority"
attribute may be alphanumeric, where "a" is a higher priority than "z", and
"1" is a higher priority than "9", for example.

 You can set a "priority" attribute in your custom processor.  However,
I would caution against absolutely relying on in-order delivery. Just
because a FlowFile begins processing first doesn't mean it will complete
first (assuming the processor has multiple concurrent tasks).  If it is
only critical that they be in order for the last processor, you might also
consider the MergeContent processor in "Defragment" mode.  Similar to the
"priority" attribute, you would set a "fragment.identifier" common to all
of the FlowFiles comprising a record, and then a "fragment.index" for each
FlowFile in the record.  At the end of the flow, you could then create a
single FlowFile comprised of all of the pieces of the record, in order.
Alternatively, you could extend the same class as MergeContent
(BinFiles[3]) in your last processor to ensure that all files are received
in order before beginning the final step.  Hope this helps

[1]
https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#connecting-components
[2]
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.MergeContent/index.html
[3]
https://github.com/apache/nifi/blob/31fba6b3332978ca2f6a1d693f6053d719fb9daa/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/BinFiles.java


Brandon


On Thu, Dec 10, 2015 at 11:37 PM, Paresh Shah 
wrote:

> Here’s my use case.
> We have a application protocol between the start and end processors in a
> data flow, that expect the flow files to arrive in the order they are
> generated. For e.g
>
> Start Record Flowfile
> 
> End Record Flowfile.
>
> The first processor does the following.
>
>   1.  Generates and transfers the StartRecord flow file.
>   2.  Generates data records and transfers them.
>   3.  Generates and transfers the EndRecord flow file
>
> The last processor in the data flow does the following.
>
>   1. Looks for the StartRecord flow file and does its thing.
>   2. Looks for the DataRecord flow file and does its thing.
>   3.  Looks for the EndRecord flow file and updates and cleanups up
> the target state.
>
> The first processor is doing multiple transfers on the session object
> before calling commit.
>
> We see that they are being received in random order. As a result we are
> not able to execute the app protocol. We have tried the
> FirstInFirstOutPrioritizer and OldestFlowFilePrioritizer.
>
> We would appreciate any insights into this we can get as it seems to be a
> blocking issue for us.
>
> Thanks
> Paresh
> 
> The information contained in this transmission may contain privileged and
> confidential information. It is intended only for the use of the person(s)
> named above. If you are not the intended recipient, you are hereby notified
> that any review, dissemination, distribution or duplication of this
> communication is strictly prohibited. If you are not the intended
> recipient, please contact the sender by reply email and destroy all copies
> of the original message.
> 
>

Re: How to dynamically update URL in GetHTTP processor ?

[Brandon DeVries]

Shweta,

 Take a look at InvokeHTTP[1] instead of GetHTTP.  InvokeHTTP allows
Expression Language in the URL, so you can specify the page number.  Let us
know if you have any other questions.  Thanks.

[1]
http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.InvokeHTTP/index.html

Brandon

On Thu, Dec 17, 2015 at 4:13 AM, shweta  wrote:

> Hi All,
>
> I have a requirement wherein I have to update  getHTTP URL dynamically.
> Within that url I have
> a parameter called page whose value can vary from 1 to N number depending
> upon no. of pages present.
>
> My requirement is as such that  "nextInt()" would not be useful. I want to
> define my own parameter like page.counter and increment it by 1 based on
> some condition. For example in following URL page=1
>
>
> https://api.stackexchange.com/2.2/posts?*page=1*&fromdate=1450224000&todate=1450310400&order=desc&sort=activity&site=stackoverflow
>
> I want to dynamically update page no. and fetch the result.
>
> Any pointers would be useful.
>
> Thanks,
> Shweta
>
>
>
>
> --
> View this message in context:
> http://apache-nifi-developer-list.39713.n7.nabble.com/How-to-dynamically-update-URL-in-GetHTTP-processor-tp5826.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>

Re: [DISCUSS] NiFi 0.5.1 release

[Brandon DeVries]

+1 for getting 0.5.1 out the door sooner rather than later.  Like you
said... if the other tickets get sorted out in a week or so, we can do
0.5.2.  Otherwise, 0.6.0 isn't that far off.  But a fix involving data loss
is worth doing quickly.

Brandon

On Fri, Feb 19, 2016 at 4:18 PM Joe Witt  wrote:

> Ok so looks like the major players for the 0.5.1 release are in.
> Today during some testing it was discovered that there can be issues
> with multiple HDFS processors with differing configurations and
> yesterday on StackOverflow it was reported that there are some issues
> talking to Kafka but there is a viable workaround for now.  I
> personally feel like what we have for an 0.5.1 is good right now
> without those two JIRAs and that we could do an 0.5.2 if those get
> sorted soon or they can go in 0.6.0 which if we stick to the
> previously discussed idea would be mid-March.
>
> What do ya'll think?
>
> If we did a kick-off of RC say late Sunday (assuming RM is on board
> too of course) we'd be looking at a midweek release for 0.5.1
>
> Thanks
> Joe
>
> On Wed, Feb 17, 2016 at 9:13 PM, Joe Witt  wrote:
> > +1
> >
> > On Wed, Feb 17, 2016 at 10:12 PM, Tony Kurc  wrote:
> >> All,
> >> Just so we're clear, my expectations is that the fixes we need above
> will
> >> go into master (0.6.0-SNAPSHOT). I (or the contributor) will attempt to
> >> also apply those fixes to a branch I made off of nifi-0.5.0, called
> >> "support/nifi-0.5.x", and when done we can test and when satisfied,
> start
> >> the 0.5.1 release process.
> >>
> >>
> >> On Wed, Feb 17, 2016 at 8:28 PM, Tony Kurc  wrote:
> >>
> >>> I lieu of a decision on a git branching strategy, does anyone object
> to me
> >>> naming a branch "support/0.5.x" so I can start pulling together 0.5.1?
> >>>
> >>> On Wed, Feb 17, 2016 at 8:17 PM, Joe Witt  wrote:
> >>>
>  Team,
> 
>  First, congrats on getting 0.5.0 out the door.  Very nice release with
>  excellent new features, enhancements, and bug fixes with involvement
>  of many new contributors.
> 
>  The effort toward 0.6.0 is well underway and if we are shooting for
>  our original target would still arrive around mid march-ish.  Seems
>  reasonable given progress so far.
> 
>  That said, I'd like to propose we go ahead and put out an 0.5.1 as
>  soon as the patches are addressed.  The finding that Lars helped
>  expose and that Mark Payne is now working  [1] in my view warrants a
>  release by itself as it is data loss related.  Also, we can address
>  the work Matt Gilman is doing to ensure properly authorized users can
>  access resources they're being blocked by [2] and the license/notice
>  findings Sean discovered during the last release [3].  There are a
>  couple odd/ends in the list as well [4].  Purely a bug fix release as
>  spec'd out in release notes [5].
> 
>  I checked with Tony and he is happy to continue performing RM tasks on
>  the 0.5.x line.
> 
>  [1] https://issues.apache.org/jira/browse/NIFI-1527
>  [2] https://issues.apache.org/jira/browse/NIFI-1497
>  [3] https://issues.apache.org/jira/browse/NIFI-1520
>  [4] https://s.apache.org/nifi-0.5.1-candidate
>  [5] https://cwiki.apache.org/confluence/display/NIFI/Release+Notes
> 
>  Thanks
>  Joe
> 
> >>>
> >>>
>

Re: [DISCUSS] git branching model

[Brandon DeVries]

I agree with Tony on  option 1.  I think it makes sense for master to be
the most "advanced" branch.  New features will then always be applied to
master, and optionally to other branches for older version support as
applicable / desired.

On Tue, Mar 29, 2016 at 10:16 AM Tony Kurc  wrote:

> I like option 1
> On Mar 29, 2016 10:03 AM, "Matt Gilman"  wrote:
>
> > Hello,
> >
> > With NiFi 0.6.0 officially released and our support strategy defined [1],
> > I'd like to revisit and propose some options for supporting both a 1.x
> > branch and 0.x branch concurrently. We need an official place where these
> > efforts can be worked, contributed to, and collaborated with the
> community.
> > I've already created a 1.x branch as a temporary place for this codebase
> to
> > live until we agree to an approach.
> >
> > Either option I'm proposing will require PRs/contributions/patches to be
> > applied to both branches as applicable. This means that the contributor
> or
> > the reviewer will need to be able to apply the commits in both places if
> > it's necessary. For instance, framework code has already started
> diverging
> > from the current master so any framework change may not need to be
> applied
> > to both if the changeset is not applicable to the 1.x baseline.
> >
> > The only question at the moment is what master will refer to.
> >
> > 1) Create a branch for 0.x and allow master to become the 1.x baseline
> > going forward. Future 0.x releases will be performed from the 0.x branch.
> > 2) Continuing working on the 1.x branch as is. Allow master to continue
> to
> > servicing 0.x releases. Once a 1.x release is made, create the 0.x branch
> > and then allow master to service 1.x releases.
> >
> > In short, when do we want master to point to the 1.x baseline? When
> should
> > we create a branch where 0.x releases will be made from. Regardless,
> > contributions will need to be performed to both places as applicable.
> >
> > Thanks.
> >
> > Matt
> >
> > [1]
> >
> >
> http://mail-archives.apache.org/mod_mbox/nifi-dev/201602.mbox/%3CCALJK9a7bWjff7xXGmUtp3nFV3HRmqbLL1StwkXcf5JdohNPRmg%40mail.gmail.com%3E
> >
> > On Sat, Feb 27, 2016 at 10:08 AM, Richard Miskin 
> > wrote:
> >
> > > I guess it will depend how much change is expected on the maintenance
> > > branches,
> > > but if you want every change in the maintenance branch to go into the
> > > main-line branch then there is little difference from a conflict point
> of
> > > view
> > > between a series of cherry-picks and a merge.
> > >
> > > Either way, it is just another approach to consider. There’s more than
> > one
> > > way to do it, and I suspect there isn’t any solution that makes it
> > trivial.
> > >
> > > Cheers,
> > > Richard
> > >
> > >
> > > > On 27 Feb 2016, at 14:43, Aldrin Piri  wrote:
> > > >
> > > > On board with Tony's points.  I think the realities of merging in
> > > practice
> > > > when that "breaking point" of sorts occurs will make the complexity
> and
> > > > overhead quite difficult and maybe even more error prone than the
> > cherry
> > > > picking approach with some additional guidelines.  When the codebase
> > > > drastically changes, the merge conflicts could be quite severe and
> > > without
> > > > a good knowledge of each part of the codebase involved during that
> > > process,
> > > > a committer may introduce regressions.
> > > >
> > > > On Sat, Feb 27, 2016 at 7:58 AM, Tony Kurc  wrote:
> > > >
> > > >> the reason I like applying patches to both lines is that once code
> > > begins
> > > >> to diverge, cleanly merging into one codebase can be impossible.
> > having
> > > >> good practices for managing patches and where they apply is
> paramount
> > > for
> > > >> success.
> > > >>
> > > >> I expect that divergence to happen with 1.x. I wanted to get in a
> > battle
> > > >> rhythm of sorts of managing multiple lines, even if the patches
> COULD
> > be
> > > >> applied to both in the manner you described.
> > > >>
> > > >> Joe W and I did a wee bit of scrambling to ensure that tickets
> marked
> > > for
> > > >> 0.5.1 had the right patches in the support branch, and some didn't,
> > so I
> > > >> think "lesson learned". I do like in the apache infrastructure that
> if
> > > >> commits have the appropriate ticket in their commit message, the
> jira
> > > will
> > > >> have the list of commits and branches those commits were applies to.
> > > >> However, I think we may need to revisit commit message "hygiene"  if
> > we
> > > >> relied on this instead of more manual review.
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> On Sat, Feb 27, 2016 at 4:45 AM, Richard Miskin <
> r.p.mis...@gmail.com
> > >
> > > >> wrote:
> > > >>
> > > >>> Hi,
> > > >>>
> > > >>> On a couple of work projects we found that the approach of
> > > cherry-picking
> > > >>> commits can lead to an unnecessarily complicated history where the
> > same
> > > >>> piece of work appears as multiple separate commits on different
> > > branches.
> > > >>> T

Re: [VOTE] Establish Apache MiNiFi, a subproject of Apache NIFi

[Brandon DeVries]

+1
On Sat, Apr 9, 2016 at 8:17 AM Mark Payne  wrote:

> +1
>
> Sent from my iPhone
>
> > On Apr 9, 2016, at 7:51 AM, Matt Gilman  wrote:
> >
> > +1
> >
> > Sent from my iPhone
> >
> >> On Apr 9, 2016, at 1:13 AM, Joe Witt  wrote:
> >>
> >> Team,
> >>
> >> Following a solid discussion [1] in early January regarding
> >> establishment of MiNiFi as a subproject of NiFi I'd like to call a
> >> formal vote to record this important community decision and
> >> established consensus.
> >>
> >> Through lazy consensus [2] the repositories were setup for source,
> >> JIRA, and git-hub mirror [3].  There has since been solid progress and
> >> discussion on a number of MINIFI JIRAs.  The core goals of MiNiFi have
> >> been documented on Confluence/Wiki [4].  It was also reported in our
> >> April 2016 board report that we have established MiNiFi based on these
> >> things.  But, lets make sure we get this right.
> >>
> >> [1] https://s.apache.org/czuz
> >> [2] https://s.apache.org/Ucxs
> >> [3]
> https://cwiki.apache.org/confluence/display/NIFI/Supporting+Infrastructure
> >> [4] https://cwiki.apache.org/confluence/display/NIFI/MiNiFi
> >>
> >> I am a +1 and am very excited by the progress made thus far!
> >>
> >> The vote will be open for 72 hours and be a majority rule vote.
> >>
> >> [ ] +1 Establish Apache MiNiFi, a subproject of Apache NiFi
> >> [ ]   0 Do not care
> >> [ ]  -1 Do not establish Apache MiNiFi
> >>
> >> Thanks
> >> Joe
>

Re: [discuss] PropertyDescriptor name and displayName attributes

[Brandon DeVries]

+1.  I think being able to move the displayName out of code an into config
/ ResourceBundle will make it much easier to support i18n[1].  If no config
is provided, the name is the default... otherwise, the name displayed is
determined by the locale.  Updating the mock framework to complain about
the absence of a ResourceBundle will encourage adoption, and we'll
gradually work our way to not being English only.

Brandon

[1] https://docs.oracle.com/javase/tutorial/i18n/intro/quick.html

On Thu, May 5, 2016 at 11:46 PM Adam Lamar  wrote:

> On Tue, May 3, 2016 at 12:09 PM, Andy LoPresto 
> wrote:
>
> > As a result of some conversations and some varying feedback on PRs, I’d
> like
> > to discuss with the community an issue I see with PropertyDescriptor name
> > and displayName attributes. I’ll describe the scenarios that cause issues
> > and my proposed solution, and then solicit responses and other
> perspectives.
>
> Andy,
>
> I'd be +1 on this as well. I think the power of this approach will
> become more clear over time, and some of the automation will make it
> possible to more widely enforce.
>
> What do you think about a mixed mode where config reading code can
> fetch the property value with either the name or display name as the
> key, defaulting to the name if it is present? A sample read of
> flow.xml.gz might look like this:
>
> * Processor asks for value of MY_CONFIG_PROPERTY
> * Configuration code looks for key "my-config-property", returns if present
> * Configuration code looks for key "My Config Property", returns if present
> * On finding no valid key, configuration returns blank/default/null
> value (whatever is done today)
>
> On configuration write, the new attributes could be written as the
> normalized name (instead of display name), to allow processors that
> have made the switch to start using the normalized name field and
> start taking advantage of the new features around it (e.g.
> internationalization). Perhaps a disadvantage to this approach is that
> it auto-upgrades an existing flow.xml.gz, making it difficult to
> downgrade from (for example) 0.7 back to 0.6.
>
> A strategy like this (or something similar) might help speed adoption
> by making the process a bit smoother for existing flow.xml.gz files
> and easier to upgrade processors incrementally. At 1.0, display names
> as configuration keys could be dropped, and the upgrade path for users
> on old 0.x releases would be to upgrade to the latest 0.x before
> making the jump to 1.0.
>
> Cheers,
> Adam
>

Re: Apache Nifi Vs Spring XD, which one is better

[Brandon DeVries]

All,

It seems like we get this sort of question a lot, and Simon's answer here
was really good.  We've had similar for discussions for Kafka[1], Storm and
Spark[2]. Should we think about adding a comparison to other technologies /
applications to the FAQ?  Not in a sales sheet sort of way, but in a way
that emphasizes how these technologies compliment each other.  Obviously we
don't need to go out and find every comparable technology, but having a
place to put answers like Simon's that are easier to reference than the
Apache mail archive might be beneficial.

Brandon

[1] https://groups.google.com/forum/#!topic/confluent-platform/JKeccNEhwaQ
[2]
http://www.zdnet.com/article/hortonworks-cto-on-apache-nifi-what-is-it-and-why-does-it-matter-to-iot/


On Fri, May 6, 2016 at 6:09 AM Simon Ball  wrote:

> ExecuteSQL can certainly deal with millions of rows. Sqoop currently makes
> more sense if you want to distribute the query processing across a large
> number of nodes (if you have 100s millions of rows 10-100GBs+ or TBs of
> data), and write direct into hadoop. If you’re looking for functionality
> like swoop’s incremental imports, then checkout QueryDatabaseTable. As long
> as you set a sensible fetch size on that (1000ish usually good, but depends
> on row size) then I’ve seen very small NiFi instances (AWS t2.small) cope
> with a few millions of rows in the order of 10 seconds.
>
> SpringXD is really a different beast to NiFi. It’s a code->deploy pattern
> rather than a command and control of data flow pattern. Once you deploy a
> SpringXD flow, it’s fixed (more like spark, storm etc compile, deploy,
> never change.) SpringXD recently added some visual design, but Flo is
> primarily a retrospective development environment (monitor a flow, not
> design it).
>
> Nifi also runs out to the edge, and gets the data. SpringXD runs in a core
> cluster (e.g. on YARN). So in this scenario, SpringXD is more like Beam or
> spark steaming. Nifi however, with site-to-site can be used to run right
> out at the edge, secure and transport data and track from origin. This
> means NiFi is actually a complement to technology like SpringXD and Beam.
> NiFi feeds these heavier weight streaming frameworks, handles the data
> movement and simple event processing, then ingesting for more complex
> analytics with the like of XD.
>
> So in short, the technologies are complementary. NiFi has the edge of
> reaching out to collect data, XD may be better for complex analytics.
>
> Simon
>
>
> > On May 6, 2016, at 6:04 AM, nehakaushik86 
> wrote:
> >
> > Hi,
> >
> > We are designing a system where we need data ingestion framework. The
> data
> > will be consumed from various data systems - DB, social feeds, text
> files,
> > CRM etc. Can you let me know how Apache Nifi fares as compared to Spring
> XD
> > and what are the best use cases where it should be used?
> >
> >
> > Also, I would like to understand the difference between Apache Nifi's
> > ExecuteSQL vs Apache Sqoop. We are planning to ingest huge amount of data
> > from DB - millions of records. Will ExecuteSQL be able to load such huge
> > volume?
> >
> >
> >
> > --
> > View this message in context:
> http://apache-nifi-developer-list.39713.n7.nabble.com/Apache-Nifi-Vs-Spring-XD-which-one-is-better-tp9963.html
> > Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
> >
>
>

Re: [discuss] PropertyDescriptor name and displayName attributes

[Brandon DeVries]

Bryan,

The tutorial I linked to[1] has a really quick view of what implementing
i18n might look like[2].  We would probably need to add "country" and
"langauge" as entries in nifi.properties, use those to create a Locale, and
then set the display name with something like:

 ResourceBundle displayNames=
ResourceBundle.getBundle("DisplayNamesBundle", currentLocale);
 String i18nName = displayNames.getString(MyProcessor.class.toString());
 PropertyDescriptor.Builder().displayName(i18nName)...

Obviously we'd want to think this through all the way, but I think this
would be the general idea.

[1] https://docs.oracle.com/javase/tutorial/i18n/intro/quick.html
[2] https://docs.oracle.com/javase/tutorial/i18n/intro/after.html


On Fri, May 6, 2016 at 9:05 AM Bryan Bende  wrote:

> I might just be resistant to change, but I am still on the fence a little
> bit...
>
> In the past the idea has always been you start out with name, and if you
> later need to change what is displayed in the UI, then you add displayName
> after the fact.
>
> It sounds like the issue is that a lot of people don't know about
> displayName, so I am totally in favor of increasing awareness through
> documentation,
> but I'm struggling with telling people that they should set displayName as
> the default behavior.
>
> For code that is contributed to NiFi, I would expect the PMC/committer
> doing the review/merging to notice if an existing property name was being
> changed and point that out in the review.
> If it was someone else's custom NAR, or even made it through the review, I
> think what happens is that the flow starts up and the processor/component
> becomes invalid because it now has an unknown property.
> This is the same behavior when we remove a property from an existing
> processor and someone upgrades, and we deemed this acceptable behavior for
> that scenario.
>
> The internationalization aspect could possibly sell me more, but I think I
> would need someone to explain how internationalization would be
> implemented, and how setting the displayName helps.
> What Brandon described makes sense to me because it is something outside
> the Java code, which is different then saying all PropertyDescriptor
> instances need name and displayName.
>
> -Bryan
>
>
> On Fri, May 6, 2016 at 8:44 AM, Brandon DeVries  wrote:
>
> > +1.  I think being able to move the displayName out of code an into
> config
> > / ResourceBundle will make it much easier to support i18n[1].  If no
> config
> > is provided, the name is the default... otherwise, the name displayed is
> > determined by the locale.  Updating the mock framework to complain about
> > the absence of a ResourceBundle will encourage adoption, and we'll
> > gradually work our way to not being English only.
> >
> > Brandon
> >
> > [1] https://docs.oracle.com/javase/tutorial/i18n/intro/quick.html
> >
> > On Thu, May 5, 2016 at 11:46 PM Adam Lamar  wrote:
> >
> > > On Tue, May 3, 2016 at 12:09 PM, Andy LoPresto 
> > > wrote:
> > >
> > > > As a result of some conversations and some varying feedback on PRs,
> I’d
> > > like
> > > > to discuss with the community an issue I see with PropertyDescriptor
> > name
> > > > and displayName attributes. I’ll describe the scenarios that cause
> > issues
> > > > and my proposed solution, and then solicit responses and other
> > > perspectives.
> > >
> > > Andy,
> > >
> > > I'd be +1 on this as well. I think the power of this approach will
> > > become more clear over time, and some of the automation will make it
> > > possible to more widely enforce.
> > >
> > > What do you think about a mixed mode where config reading code can
> > > fetch the property value with either the name or display name as the
> > > key, defaulting to the name if it is present? A sample read of
> > > flow.xml.gz might look like this:
> > >
> > > * Processor asks for value of MY_CONFIG_PROPERTY
> > > * Configuration code looks for key "my-config-property", returns if
> > present
> > > * Configuration code looks for key "My Config Property", returns if
> > present
> > > * On finding no valid key, configuration returns blank/default/null
> > > value (whatever is done today)
> > >
> > > On configuration write, the new attributes could be written as the
> > > normalized name (instead of display name), to allow processors that
> > > have made the switch to start using the normalized name field and
> > 

Re: [discuss] PropertyDescriptor name and displayName attributes

[Brandon DeVries]

+1.  I like that better.  Deprecate displayName(), and set it
"automatically" based on the locale from properties.  The name of the
property (which should never change) is the key into the ResourceBundle.

Brandon


On Fri, May 6, 2016 at 9:24 AM Matt Burgess  wrote:

> Same here. Internationalization is often implemented as properties
> files/resources, where you possibly load in a file based on the system
> setting for Locale (like processor_names_en_US.properties). If we were
> to do internationalization this way (i.e. a non-code based solution,
> which is more flexible), then ironically displayName() might/should be
> deprecated in favor of using the value of name() as the key in a
> properties/lookup file; the corresponding value would be the
> appropriate locale-specific "display name".
>
> Brandon's links show this approach, I have seen this i18n approach on
> other projects/products and it seems to work pretty well.
>
> Regards,
> Matt
>
> On Fri, May 6, 2016 at 9:11 AM, Joe Witt  wrote:
> > I share Bryan's perspective.
> >
> > On Fri, May 6, 2016 at 9:05 AM, Bryan Bende  wrote:
> >> I might just be resistant to change, but I am still on the fence a
> little
> >> bit...
> >>
> >> In the past the idea has always been you start out with name, and if you
> >> later need to change what is displayed in the UI, then you add
> displayName
> >> after the fact.
> >>
> >> It sounds like the issue is that a lot of people don't know about
> >> displayName, so I am totally in favor of increasing awareness through
> >> documentation,
> >> but I'm struggling with telling people that they should set displayName
> as
> >> the default behavior.
> >>
> >> For code that is contributed to NiFi, I would expect the PMC/committer
> >> doing the review/merging to notice if an existing property name was
> being
> >> changed and point that out in the review.
> >> If it was someone else's custom NAR, or even made it through the
> review, I
> >> think what happens is that the flow starts up and the
> processor/component
> >> becomes invalid because it now has an unknown property.
> >> This is the same behavior when we remove a property from an existing
> >> processor and someone upgrades, and we deemed this acceptable behavior
> for
> >> that scenario.
> >>
> >> The internationalization aspect could possibly sell me more, but I
> think I
> >> would need someone to explain how internationalization would be
> >> implemented, and how setting the displayName helps.
> >> What Brandon described makes sense to me because it is something outside
> >> the Java code, which is different then saying all PropertyDescriptor
> >> instances need name and displayName.
> >>
> >> -Bryan
> >>
> >>
> >> On Fri, May 6, 2016 at 8:44 AM, Brandon DeVries  wrote:
> >>
> >>> +1.  I think being able to move the displayName out of code an into
> config
> >>> / ResourceBundle will make it much easier to support i18n[1].  If no
> config
> >>> is provided, the name is the default... otherwise, the name displayed
> is
> >>> determined by the locale.  Updating the mock framework to complain
> about
> >>> the absence of a ResourceBundle will encourage adoption, and we'll
> >>> gradually work our way to not being English only.
> >>>
> >>> Brandon
> >>>
> >>> [1] https://docs.oracle.com/javase/tutorial/i18n/intro/quick.html
> >>>
> >>> On Thu, May 5, 2016 at 11:46 PM Adam Lamar 
> wrote:
> >>>
> >>> > On Tue, May 3, 2016 at 12:09 PM, Andy LoPresto  >
> >>> > wrote:
> >>> >
> >>> > > As a result of some conversations and some varying feedback on
> PRs, I’d
> >>> > like
> >>> > > to discuss with the community an issue I see with
> PropertyDescriptor
> >>> name
> >>> > > and displayName attributes. I’ll describe the scenarios that cause
> >>> issues
> >>> > > and my proposed solution, and then solicit responses and other
> >>> > perspectives.
> >>> >
> >>> > Andy,
> >>> >
> >>> > I'd be +1 on this as well. I think the power of this approach will
> >>> > become more clear over time, and some of the automation will make it
> >>> > possible to more widely enforce.
> >>&g

Re: [discuss] PropertyDescriptor name and displayName attributes

[Brandon DeVries]

I guess my only objection is, do we really need Option 2?  If all you want
to do is provide a name, fine.  But, if you want to change the name (for
better communicating the purpose to the user, be that in English, French,
or Esperanto) option 3 provides the mechanism for that.  It's not as easy
for the developer as option 2, but it has the benefit of starting the
process of creating resource bundles for different languages (and
potentially deprecating a semi-confusing option).  This means that someone
wanting to add support for a different language (e.g. french) has a lower
barrier to entry, in that they can just find the "displayNames.properties"
file, and put a "displayNames_fr_FR.properties next to it.  Ultimately its
a trade off of conveniences... the developer wanting to change a property
name vs. a potential contributor wanting to improve language support.  I
guess I would argue that this shouldn't be that hard for a developer to
figure out, and lean in favor of encouraging language contributions.

Ultimately, it isn't' that strong an objection though, and it does sound
like this is heading in the right direction...

Brandon

On Fri, May 6, 2016 at 10:25 AM Joe Witt  wrote:

> Definitely on board with the idea that the 'name' will be the key to a
> resource bundle.  It does imply such names will need to follow
> necessary conventions to be valid resource bundle keys.
>
> However, in the spirit of always thinking about the developer path to
> productivity I am hopeful we can come up with a nice way to not
> require them to setup a resource bundle.
>
> The idea being that the following order of operations/thought would exist:
>
> 1) How can I provide a new property to this processor?
> Answer: Add a property descriptor and set the name.  This name will be
> used to refer to the property descriptor whenever serialized/saving
> the config and it will be rendered through the REST API and thus made
> available as the property name in the UI.
>
> 2) Oh snap.  I wish I had used a different name because I've found a
> better way to communicate intent to the user.  How do I do this?
> Answer: Go ahead and set displayName.  NiFi will continue to use the
> 'name' for serialization/config saving but will use the displayName
> for what is shown to the user in the UI.
>
> 3) I would like to support locale sensitive representations of my
> property name.  How can I do this?
> Answer: Add a resource bundle with entries for your property 'name'
> value.  This means the resource bundle needs to exist and your
> property 'name' must adhere to resource bundle key naming requirements
> [1].  If this is supplied and can be looked up then this will be used
> and otherwise will fallback to using displayName value if present and
> otherwise will fallback to using the value of 'name'.
>
> And in any event we still need to better document/articulate this
> model as the root of this thread was that we hadn't effectively
> communicated the existence of displayName.  I agree this discussion
> ended up getting us to a great place though as we should all strive to
> support internationalization.
>
> With an approach like this I am onboard.  I think this balances our
> goals of having a simple to use API but also allows those who want to
> support multiple locales to do so cleanly.
>
> Thanks
> Joe
>
> [1] https://docs.oracle.com/javase/tutorial/i18n/resbundle/propfile.html
>
> On Fri, May 6, 2016 at 9:33 AM, Brandon DeVries  wrote:
> > +1.  I like that better.  Deprecate displayName(), and set it
> > "automatically" based on the locale from properties.  The name of the
> > property (which should never change) is the key into the ResourceBundle.
> >
> > Brandon
> >
> >
> > On Fri, May 6, 2016 at 9:24 AM Matt Burgess  wrote:
> >
> >> Same here. Internationalization is often implemented as properties
> >> files/resources, where you possibly load in a file based on the system
> >> setting for Locale (like processor_names_en_US.properties). If we were
> >> to do internationalization this way (i.e. a non-code based solution,
> >> which is more flexible), then ironically displayName() might/should be
> >> deprecated in favor of using the value of name() as the key in a
> >> properties/lookup file; the corresponding value would be the
> >> appropriate locale-specific "display name".
> >>
> >> Brandon's links show this approach, I have seen this i18n approach on
> >> other projects/products and it seems to work pretty well.
> >>
> >> Regards,
> >> Matt
> >>
> >> On Fri, May 6, 2016 at 9:11

Re: [discuss] PropertyDescriptor name and displayName attributes

[Brandon DeVries]

With 1.0.0, could we deprecate name() and replace it with id()?  Under the
hood name() would still call id() (which would just be the current name()
renamed...) until we finally pull the plug on it, but in the meantime it
might encourage developers to understand what name / id really means.

Brandon

On Fri, May 6, 2016 at 10:34 AM Oleg Zhurakousky <
ozhurakou...@hortonworks.com> wrote:

> I think the the main source of confusion (as it was for me) is the ‘name’
> name itself.
> Basically the ‘name’ today is really an identifier of the property and
> therefore could/should never change while 'displayName' is volatile.
> Obviously changing “name’ to “id” is out of the question as it would break
> practically everything. But from the documentation perspective may be we
> should consider documenting that ‘name’ corresponds to a unique and
> perpetual identifier of the property that would also be displayed unless
> override by ‘displayName’ while such override would only affect the display
> characteristics of the property and not its identification.
>
> Cheers
> Oleg
>
> > On May 6, 2016, at 10:25 AM, Joe Witt  wrote:
> >
> > Definitely on board with the idea that the 'name' will be the key to a
> > resource bundle.  It does imply such names will need to follow
> > necessary conventions to be valid resource bundle keys.
> >
> > However, in the spirit of always thinking about the developer path to
> > productivity I am hopeful we can come up with a nice way to not
> > require them to setup a resource bundle.
> >
> > The idea being that the following order of operations/thought would
> exist:
> >
> > 1) How can I provide a new property to this processor?
> > Answer: Add a property descriptor and set the name.  This name will be
> > used to refer to the property descriptor whenever serialized/saving
> > the config and it will be rendered through the REST API and thus made
> > available as the property name in the UI.
> >
> > 2) Oh snap.  I wish I had used a different name because I've found a
> > better way to communicate intent to the user.  How do I do this?
> > Answer: Go ahead and set displayName.  NiFi will continue to use the
> > 'name' for serialization/config saving but will use the displayName
> > for what is shown to the user in the UI.
> >
> > 3) I would like to support locale sensitive representations of my
> > property name.  How can I do this?
> > Answer: Add a resource bundle with entries for your property 'name'
> > value.  This means the resource bundle needs to exist and your
> > property 'name' must adhere to resource bundle key naming requirements
> > [1].  If this is supplied and can be looked up then this will be used
> > and otherwise will fallback to using displayName value if present and
> > otherwise will fallback to using the value of 'name'.
> >
> > And in any event we still need to better document/articulate this
> > model as the root of this thread was that we hadn't effectively
> > communicated the existence of displayName.  I agree this discussion
> > ended up getting us to a great place though as we should all strive to
> > support internationalization.
> >
> > With an approach like this I am onboard.  I think this balances our
> > goals of having a simple to use API but also allows those who want to
> > support multiple locales to do so cleanly.
> >
> > Thanks
> > Joe
> >
> > [1] https://docs.oracle.com/javase/tutorial/i18n/resbundle/propfile.html
> >
> > On Fri, May 6, 2016 at 9:33 AM, Brandon DeVries  wrote:
> >> +1.  I like that better.  Deprecate displayName(), and set it
> >> "automatically" based on the locale from properties.  The name of the
> >> property (which should never change) is the key into the ResourceBundle.
> >>
> >> Brandon
> >>
> >>
> >> On Fri, May 6, 2016 at 9:24 AM Matt Burgess 
> wrote:
> >>
> >>> Same here. Internationalization is often implemented as properties
> >>> files/resources, where you possibly load in a file based on the system
> >>> setting for Locale (like processor_names_en_US.properties). If we were
> >>> to do internationalization this way (i.e. a non-code based solution,
> >>> which is more flexible), then ironically displayName() might/should be
> >>> deprecated in favor of using the value of name() as the key in a
> >>> properties/lookup file; the corresponding value would be the
> >>> appropriate locale-specific "display na

Re: [discuss] PropertyDescriptor name and displayName attributes

[Brandon DeVries]

Joe,

Alright, I buy that.  As a community, we try not to do #2 (stop
laughing...). But we still allow someone writing / maintaining custom
components to do so without figuring out how to set up a resource bundle.
+1.

Brandon

On Fri, May 6, 2016 at 10:51 AM Joe Witt  wrote:

> Brandon
>
> Understood.  I'd say I lean more towards providing 1, 2, and 3 because
> allows us to be flexible in what we accept and conservative in what we
> do.  By that I mean we as a community can go ahead and enforce that we
> basically just adopt 3 and use RTC to help enforce that.  But that
> from an API and doing examples perspective we allow 1 and 2 as well.
> I don't think it adds much weight but does add flexibility.
>
> Like you, I think I'm ok either way.  This was a good discussion and
> ending up with supporting internationalization and flexibility in
> displayed name while retaining configuration sanity - is a win.
>
> Thanks
> Joe
>
> On Fri, May 6, 2016 at 10:45 AM, Brandon DeVries  wrote:
> > I guess my only objection is, do we really need Option 2?  If all you
> want
> > to do is provide a name, fine.  But, if you want to change the name (for
> > better communicating the purpose to the user, be that in English, French,
> > or Esperanto) option 3 provides the mechanism for that.  It's not as easy
> > for the developer as option 2, but it has the benefit of starting the
> > process of creating resource bundles for different languages (and
> > potentially deprecating a semi-confusing option).  This means that
> someone
> > wanting to add support for a different language (e.g. french) has a lower
> > barrier to entry, in that they can just find the
> "displayNames.properties"
> > file, and put a "displayNames_fr_FR.properties next to it.  Ultimately
> its
> > a trade off of conveniences... the developer wanting to change a property
> > name vs. a potential contributor wanting to improve language support.  I
> > guess I would argue that this shouldn't be that hard for a developer to
> > figure out, and lean in favor of encouraging language contributions.
> >
> > Ultimately, it isn't' that strong an objection though, and it does sound
> > like this is heading in the right direction...
> >
> > Brandon
> >
> > On Fri, May 6, 2016 at 10:25 AM Joe Witt  wrote:
> >
> >> Definitely on board with the idea that the 'name' will be the key to a
> >> resource bundle.  It does imply such names will need to follow
> >> necessary conventions to be valid resource bundle keys.
> >>
> >> However, in the spirit of always thinking about the developer path to
> >> productivity I am hopeful we can come up with a nice way to not
> >> require them to setup a resource bundle.
> >>
> >> The idea being that the following order of operations/thought would
> exist:
> >>
> >> 1) How can I provide a new property to this processor?
> >> Answer: Add a property descriptor and set the name.  This name will be
> >> used to refer to the property descriptor whenever serialized/saving
> >> the config and it will be rendered through the REST API and thus made
> >> available as the property name in the UI.
> >>
> >> 2) Oh snap.  I wish I had used a different name because I've found a
> >> better way to communicate intent to the user.  How do I do this?
> >> Answer: Go ahead and set displayName.  NiFi will continue to use the
> >> 'name' for serialization/config saving but will use the displayName
> >> for what is shown to the user in the UI.
> >>
> >> 3) I would like to support locale sensitive representations of my
> >> property name.  How can I do this?
> >> Answer: Add a resource bundle with entries for your property 'name'
> >> value.  This means the resource bundle needs to exist and your
> >> property 'name' must adhere to resource bundle key naming requirements
> >> [1].  If this is supplied and can be looked up then this will be used
> >> and otherwise will fallback to using displayName value if present and
> >> otherwise will fallback to using the value of 'name'.
> >>
> >> And in any event we still need to better document/articulate this
> >> model as the root of this thread was that we hadn't effectively
> >> communicated the existence of displayName.  I agree this discussion
> >> ended up getting us to a great place though as we should all strive to
> >> support internationalization.
> >>
> >> With an appro

Re: [DISCUSS] Removal of FlowFilePrioritizer as first-class extension point

[Brandon DeVries]

+1.  This seems like something we should provide options for (as we do),
but doesn't really need to be made / kept accessible for extension.

Brandon

On Fri, May 6, 2016 at 11:45 AM Mark Payne  wrote:

> I'm definitely a +1. In my experience, the way that most people think
> about prioritizing data is
> to either assign an absolute priority to a FlowFile and use the
> PriorityAttributePrioritizer or to
> use the FirstInFirstOut Prioritizer. Any number of processors can be used
> to extract the the
> 'priority' attribute and prioritize the data that way. I think this makes
> the extensibility less valuable,
> since the flow itself can be used to determine a 'priority' attribute
> based on FlowFile content, attributes,
> etc.
>
> > On May 6, 2016, at 11:16 AM, Joe Witt  wrote:
> >
> > Team,
> >
> > I'd like to propose we remove the FlowFilePrioritizer [1] from the set
> > of first class extension points we support.
> >
> > The background:
> >
> > FlowFilePrioritizer implementations are used to compare flow files as
> > they are enqueued on a given connection in the flow.  This in turn
> > means when flow files are pulled from the queue they are pulled in a
> > manner that allows the most important data first to be operated on.
> > This is a valuable feature and is heavily utilized.  Out of the box
> > NiFi provides several obvious prioritizer implementations such as
> > first in and out based on age of the flow file, first in based on
> > entry order, and honoring a numeric representation of priority set as
> > a specific attribute [2].  They are rarely changed and have so far not
> > grown in numbers nor have there been any discussions of doing so.  If
> > I think back to their usage over the past decade I actually think
> > there have been only a few ever made.
> >
> > The concept and ability to sort queues is important and powerful and
> > needs to be kept.  But making them a first-class extension point I am
> > now questioning the value of.  The reason being is that as defined the
> > interface is intuitive for the developer but much harder for the
> > framework side.  That combined with their lack of ever being extended
> > opens the debate.
> >
> > When the prioritizers were first envisioned we didn't support the
> > concept of swapping out flowfiles to disk when the queues were huge.
> > We now do.  But we cannot sort (at this time) the swapped out items.
> > By getting rid of this extension point as it is now we can instead
> > support these types of prioritizers in a different and more optimized
> > manner albeit in a less extension friendly way (more coupled to the
> > framework).  Rather than simply using comparators we can do absolute
> > priority assignment and when swapping out flow files we can track the
> > largest/smallest priority and thus enable prioritized swap-in.  This
> > would also be helpful for doing things like auto-cluster load
> > balancing or cluster-wide prioritized site-to-site.
> >
> > So, in short, the interface would go from being a comparator to
> > instead providing a method which returns an absolute priority.  For
> > example, it would have a method called 'getPriority' which takes in a
> > flow file and returns a long.
> >
> > This approach would also still allow chaining prioritizers as we do
> today.
> >
> > We still can support this as something which can be extended for those
> > who wish to do so just in a less friendly and more framework coupled
> > manner.  Basically, this would just be more like we support content
> > repository or provenance repository extension where the developer
> > needs to both understand the implementation they want but also the
> > mechanics of getting that into the build and the deeper implications.
> >
> > Would like to hear if others are supportive of this or if they see any
> > major problems posed by this.  Given we're working towards the 1.x
> > release this is a good time to pull this cord.  If we do this we can
> > document the steps and thinking needed to build/contribute new
> > prioritizer schemes.
> >
> > Thanks
> > Joe
> >
> > [1]
> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=blob;f=nifi-api/src/main/java/org/apache/nifi/flowfile/FlowFilePrioritizer.java;h=684f454f57094a0e1f669333d63be06cd5a8a043;hb=refs/heads/0.x
> > [2]
> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=tree;f=nifi-nar-bundles/nifi-standard-bundle/nifi-standard-prioritizers/src/main/java/org/apache/nifi/prioritizer;h=6d5db994f9fd9624bf7f548ebd69548b6917ccd1;hb=refs/heads/0.x
>
>

Re: [DISCUSS] Extraction of Extension API

[Brandon DeVries]

+1.  Seems like a good idea, and now is a good time.

Brandon

On Fri, May 6, 2016 at 3:31 PM Aldrin Piri  wrote:

> All,
>
> I would like to propose a refactoring of the nifi-api for our master/1.0
> branch.  In summary, a lightweight and concise view of this module allows
> for reduced footprint of the NIFI API for components and minimizes the
> creep of those items that authoring components do not need to use.
>
> In a broader context there is a core set of interfaces that users will
> interface with in their generation of new extensions for NiFi.  Summarily,
> these components have comprised Processors, Controller Services, Reporting
> Tasks, & Prioritizers (the last of which is currently under discussion to
> potentially be removed from this forward facing status).
>
> What I would like to suggest is the refactoring of the nifi-api module to
> be broken down into to two components: the nifi-api and the
> nifi-framework-api.  nifi-api will encompass all things developers would
> need to provide extensions tailored toward interacting with dataflow.
>  nifi-framework-api would address the more internal items that are also
> extensible, but not something that is typically implemented and would
> consist of the remainder of those items not in nifi-api.
>
> This enables a core set of APIs that extensions can implement with a
> broader perspective of providing API compatibility between both NiFi and
> MiNiFi.  This enables some nice reuse of work with the goal ultimately
> amounting to, write for NiFi, run on MiNiFi and the reverse.
>
> Logistically, for NiFi extension writers, at first glance, not much would
> change with their efforts.  The core dependency would remain the same, but
> would be curtailed in its scope to only what is required.  Framework
> components of course, would need to be updated to include a dependency on
> nifi-framework-api.
>
> Given that the new structure would not yet be released, and to allow MiNiFi
> to continue on its development path, a Git submodule or subtree would be
> introduced into MiNiFi and supporting documentation on how to make use of
> this for those not familiar.
>

Re: [VOTE] Release Apache NiFi 0.7.0 (RC1)

[Brandon DeVries]

In that case, I'd like to request the vote be extended to Tuesday
afternoon. There is at least one critical bug fix that just got in that we
haven't been able to test. We've only been able to reliably replicate the
problem on one system, and I do not have access to it over the weekend. It
is not impossible that we could confirm by Monday afternoon, but a number
of things would have to go right for it to happen.

Brandon

On Sat, Jun 25, 2016 at 9:13 AM Joe Witt  wrote:

> Timing these is not easy.  But requesting a courtesy weekend extra day is
> totally reasonable and has been done in the past.
> On Jun 25, 2016 8:09 AM, "Ryan H"  wrote:
>
> > I'd love to get the opportunity to test the nifi-0.7.0-RC1 tag on my
> > integration server at work.. Seeing as though the voting started Friday
> at
> > 3:34.. and ends Monday at 3:34, I'm not sure I'll get the opportunity.
> >
> > Recommend in the future this is done mon-fri vs the weekend.
> >
> > Ryan
> >
> > On Sat, Jun 25, 2016 at 10:39 AM, Joe Witt  wrote:
> >
> > > Not quite ready to vote but here is the status of my review.
> > >
> > > Checked out the build/commit/hashes/etc.. and all those things checked
> > out.
> > >
> > > Extraneous file:
> > > - It has /appveyor.yml file.  Obviously this is not a big deal but
> > > should be ignored from source release somehow.
> > > Recommendation: File a JIRA to to block this from being in the source
> > > release.
> > >
> > > Licensing concerns
> > > - The LICENSE file has this entry "This product bundles 'Jolt' which
> > > is available under an Apache License"
> > >   We should not have apache licensed items in our license file. We
> > > only need to carry forward NOTICE information of such dependencies and
> > > those would go in our NOTICE if applicable.
> > > - The nifi-assembly/NOTICE file has an "MIT License" section.  This
> > > should be removed.  MIT licensed items should be reflected in the
> > > LICENSE file and Luaj already appears to be there.
> > > - The nifi-assembly/NOTICE file references "Python Software
> > > Foundation" section.  Python license is category A.  Such dependency
> > > belongs in LICENSE file and not NOTICE.  Only if that dependency has a
> > > NOTICE with legally required items should there be a reference to it.
> > > Recommendation: File a JIRA to correct these items.
> > >
> > > Functional issue
> > > - Cannot get a secured cluster working that worked well on 0.6.0.
> > > After upgrading now seeing the following line.  It either means I
> > > upgraded incorrectly, or we're missing critical migration guidance, or
> > > we have introduced a new bug.  I cannot login to the cluster and in
> > > the nifi-user.log of the NCM if found this entry:
> > >   2016-06-25 14:19:12,017 INFO [NiFi Web Server-23]
> > > o.a.n.w.a.c.IllegalArgumentExceptionMapper
> > > java.lang.IllegalArgumentException: User account already created
> > > CN=box1.testing.org, OU=NIFI, O=Apache-NiFi, L=Here, ST=There,
> > > C=EVERYWHERE. Returning Bad Request response.
> > > Status: Still looking into this.  Do not yet understand what is
> > happening.
> > >
> > > We presently do not have migration guidance for moving from 0.6 to
> > > 0.7.  This may be correct but wanted to verify.
> > >
> > > Thanks
> > > Joe
> > >
> > > On Sat, Jun 25, 2016 at 6:55 AM, Pierre Villard
> > >  wrote:
> > > > Hi,
> > > >
> > > > +1
> > > >
> > > > Thanks Joe for the amazing work to have this RC out.
> > > >
> > > > Ran through the helper release guide.
> > > > Signature and checksums are OK.
> > > > Maven build with tests and contrib-check OK on Windows 10.
> > > > Tested some flows and everything looks OK.
> > > >
> > > > -Pierre
> > > >
> > > >
> > > > 2016-06-24 21:34 GMT+02:00 Joe Percivall
> >  > > >:
> > > >
> > > >> Hello Apache NiFi Community,
> > > >>
> > > >> I am pleased to be calling this vote for the source release of
> Apache
> > > NiFi,
> > > >> nifi-0.7.0.
> > > >>
> > > >> The source zip, including signatures, digests, etc. can be found at:
> > > >>
> > https://repository.apache.org/content/repositories/orgapachenifi-1083/
> > > >>
> > > >> The Git tag is nifi-0.7.0-RC1
> > > >> The Git commit hash is 56dc8aefac822454f2d4133e8cbde903a628ec5b
> > > >> *
> > > >>
> > >
> >
> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=56dc8aefac822454f2d4133e8cbde903a628ec5b
> > > >> *
> > > >>
> > >
> >
> https://github.com/apache/nifi/commit/56dc8aefac822454f2d4133e8cbde903a628ec5b
> > > >>
> > > >> Checksums of nifi-0.7.0-source-release.zip:
> > > >> MD5: 8dc31d1f8103f9679cfe4b9533a4049b
> > > >> SHA1: dc160c9611dc18523c994cd9086fc7088cc520f4
> > > >> SHA256:
> > 7df056fb6af0eb9bdb94c17bf5bc2db288ed2b0d0a8ad23127dab5cdf74ce767
> > > >>
> > > >> Release artifacts are signed with the following key:
> > > >> http://pgp.mit.edu/pks/lookup?op=get&search=0x3C6DBAFE22886FEE
> > > >>
> > > >> KEYS file available here:
> > > >> https://dist.apache.org/repos/dist/release/nifi/KEYS
> > > >>
> > > >> 131 issues were closed/resol

Re: [DISCUSS] Closing in on a 0.x release

[Brandon DeVries]

I agree sooner rather than later for cutting 0.7.1. I think Mike's question
to some degree was whether or not some of those tickets were worth fixing
in 0.x. For example, I'm not sure how much I care about:

NIFI-2571 deprecate NiFiProperties.getInstance()
NIFI-2163 nifi.sh follow the Linux service spec

On the other, there are some I would like to see, even if its in 0.7.2 or
0.8.0, e.g.:

NIFI-2433 "Primary Node Only" processors
NIFI-2562 PutHDFS data corruption

But, there are a number of things that are currently committed (or have
patch available) that I'd like to see available as soon as possible. So
rather than wait for more "nice to haves", I'd rather address the immediate
needs... Immediately.

Brandon


On Tue, Sep 27, 2016 at 10:15 PM Tony Kurc  wrote:

> I think I brought this up before, I sort of expected we may do more 0.x
> releases. I certainly think the more the bugs we can fix, the merrier, and
> it seems like your list is a good initial strawman for a bug fix release of
> we collectively would like to put one together.
>
> While the tickets with work to do on them would be great to have fixed, I
> personally would rather see a release with some fixes and a couple known
> issues than holding off for "perfection", especially as a lot of our effort
> is on 1.x. Are you asking if effort would be wasted if patches were
> developed for the 0.x issues?
>
> Fwiw, I certainly could do the RM work if there is interest/demand signal
> for in another 0.x.
>
> On Sep 27, 2016 5:28 PM, "Michael Moser"  wrote:
>
> > All,
> >
> > I would like to start the discussion of making the next official release
> of
> > the 0.x branch.  I propose that this release be numbered 0.7.1 since it
> > seems that only bug fixes have occurred on the 0.x branch since 0.7.0 was
> > released.
> >
> > The JIRA link [1] below can show you the tickets that have been completed
> > in the 0.x branch.  There are 33 tickets in this list that are resolved.
> >
> > Here is a list of JIRA tickets that are not yet complete that we need to
> > decide what to do with.
> >
> > Patch Available
> > NIFI-2429 PersistentProvenanceRepository
> > NIFI-2774 ConsumeJMS
> >
> > Open against 0.7.0
> > NIFI-2383 ListFiles
> > NIFI-2433 "Primary Node Only" processors (fixed in master but this ticket
> > is for 0.x)
> > NIFI-2798 Zookeeper security upgrade
> > NIFI-2801 Kafka processors documentation
> >
> > Other high priority bugs not yet specifically targeted to the 0.x branch,
> > should we try to work these?
> > NIFI-1696 Event Driven processors
> > NIFI-1912 PutEmail content-type
> > NIFI-2163 nifi.sh follow the Linux service spec
> > NIFI-2409 StoreKiteInDataset invalid URI
> > NIFI-2562 PutHDFS data corruption
> > NIFI-2571 deprecate NiFiProperties.getInstance()
> >
> > -- Mike
> >
> > [1] -
> > https://issues.apache.org/jira/browse/NIFI-2801?jql=
> > project%20%3D%20NIFI%20AND%20fixVersion%20in%20%280.7.1%2C%200.8.0%29
> >
>