RE: [EXT] Re: NiFi 1.2.0 Record processors question

[Peter Wicks (pwicks)]

I appreciate the clarification as well. I was really confused why my Avro files 
weren't converting, and this explains it; though I have to say the error 
messages you run into in this scenario are not clear.

-Original Message-
From: Koji Kawamura [mailto:ijokaruma...@gmail.com] 
Sent: Monday, May 22, 2017 9:59 AM
To: dev 
Subject: [EXT] Re: NiFi 1.2.0 Record processors question

I've updated the JIRA description to cover not only embedded Avro schema but 
also ones such as derived from CSVReader.
https://issues.apache.org/jira/browse/NIFI-3921

Thanks,
Koji

On Sat, May 20, 2017 at 4:14 AM, Joe Gresock  wrote:
> Yes, both of your examples help explain the use of the CSV header parsing.
>
> I think I have a much better understanding of the new framework now, 
> thanks to Bryan and Matt.  Mission accomplished!
>
> On Fri, May 19, 2017 at 7:04 PM, Bryan Bende  wrote:
>
>> When a reader produces a record it attaches the schema it used to the 
>> record, but we currently don't have a way for a writer to use that 
>> schema when writing a record, although I think we do want to support 
>> that... something like a "Use Schema in Record" option as a choice in 
>> the 'Schema Access Strategy' of writers.
>>
>> For now, when a processor uses a reader and a writer, and you also 
>> want to read and write with the same schema, then you would still 
>> have to define the same schema for the writer to use even if you had 
>> a CSV reader that inferred the schema from the headers.
>>
>> There are some processors that only use a reader, like 
>> PutDabaseRecord, where using the CSV header would still be helpful.
>>
>> There are also a lot of cases where you where you would write with a 
>> different schema then you read with, so using the CSV header for 
>> reading is still helpful in those cases too.
>>
>> Hopefully I am making sense and not confusing things more.
>>
>>
>> On Fri, May 19, 2017 at 1:27 PM, Joe Gresock  wrote:
>> > Matt,
>> >
>> > Great response, this does help explain a lot.  Reading through your 
>> > post made me realize I didn't understand the AvroSchemaRegistry.  I 
>> > had been thinking it was something that nifi processors populated, 
>> > but I re-read
>> its
>> > usage description and it does indeed say to use dynamic properties 
>> > for
>> the
>> > schema name / value.  In that case, I can definitely see how this 
>> > is not dynamic in the sense of inferring any schemas on the flow.  
>> > It makes me wonder if there would be a way to populate the schema 
>> > registry from flow files.  When I first glanced at the processors, 
>> > I had assumed that when
>> the
>> > schema was inferred from the CSV headers, it would create an entry 
>> > in the AvroSchemaRegistry, provided you filled in the correct properties.
>> Clearly
>> > this is not how it works.
>> >
>> > Just so I understand, does your first paragraph mean that even if 
>> > you use the CSV headers to determine the schema, you still can't 
>> > use the *Record processors unless you manually register a matching 
>> > schema in the schema registry, or otherwise somehow set the schema 
>> > in an attribute?  In this case, it almost seems like inferring the 
>> > schema from the CSV headers
>> serves
>> > no purpose, and I don't see how NIFI-3921 would alleviate that (it 
>> > only appears to address avro flow files with embedded schema).
>> >
>> > Based on this understanding, I was able to successfully get the 
>> > following flow working:
>> > InferAvroSchema -> QueryRecord.
>> >
>> > QueryRecord uses CSVReader with "Use Schema Text Property" and 
>> > Schema
>> Text
>> > set to ${inferred.avro.schema} (which is populated by the 
>> > InferAvroSchema processor).  It also uses JsonRecordSetWriter with 
>> > a similar configuration.  I could attach a template, but I don't 
>> > know the best way
>> to
>> > do that on the listserve.
>> >
>> > Joe
>> >
>> > On Fri, May 19, 2017 at 4:59 PM, Matt Burgess 
>> > 
>> wrote:
>> >
>> >> Joe,
>> >>
>> >> Using the CSV Headers to determine the schema is currently the 
>> >> only "dynamic" schema strategy, so it will be tricky to use with 
>> >> the other Readers/Writers and associated processors (which require 
>> >> an explicit schema). This should be alleviated with NIFI-3291 [1].  
>> >> For this first release, I believe the approach would be to 
>> >> identify the various schemas for your incoming/outgoing data, 
>> >> create a Schema Registry with all of them, then the various Record 
>> >> Readers/Writers using those.
>> >>
>> >> There were some issues during development related to using the 
>> >> incoming vs outgoing schema for various record operations, if 
>> >> QueryRecord seems to be using the output schema for querying then 
>> >> it is likely a bug. However in this case it might just be that you 
>> >> need an explicit schema for your Writer that matches the input 
>> >> schema (which is inferred from the CSV header). The CSV Header 
>> >> inference currently assumes all field

Re: Contributing templates

[Joe Witt]

Both Pierre and Joe can contribute on the wiki now.

Thanks!

On Mon, May 22, 2017 at 9:26 PM, Joe Gresock  wrote:
> Thanks Pierre!
>
> So, now that you are PMC, what say you? ;)
>
> 
> Virus-free.
> www.avast.com
> 
> <#m_4722898169785819424_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>
> On Sat, May 20, 2017 at 11:50 AM, Pierre Villard <
> pierre.villard...@gmail.com> wrote:
>
>> Hey Joe,
>>
>> I believe there is a confluence page with examples [1] that could be a good
>> place but you would need write access on the page or to ask someone with
>> write privileges. Or you could add your template as a comment on this page
>> I guess. I'll let PMC say what is best ;-)
>>
>> [1]
>> https://cwiki.apache.org/confluence/display/NIFI/
>> Example+Dataflow+Templates
>>
>> Pierre
>>
>> 2017-05-20 16:55 GMT+02:00 Joe Gresock :
>>
>> > All,
>> >
>> > Is there a place to contribute templates?  I have a template for using
>> the
>> > SiteToSiteStatusReportingTask along with ConvertRecord and
>> > PutElasticsearchHttp that automatically stores NiFi's status history in
>> an
>> > Elasticsearch database for long term analysis of NiFi status history.
>> >
>> > If there is a place for contributions like this, what is the best way to
>> > upload the template/description?
>> >
>> > --
>> > I know what it is to be in need, and I know what it is to have plenty.  I
>> > have learned the secret of being content in any and every situation,
>> > whether well fed or hungry, whether living in plenty or in want.  I can
>> do
>> > all this through him who gives me strength.*-Philippians 4:12-13*
>> >
>>
>
>
>
> --
> I know what it is to be in need, and I know what it is to have plenty.  I
> have learned the secret of being content in any and every situation,
> whether well fed or hungry, whether living in plenty or in want.  I can do
> all this through him who gives me strength.*-Philippians 4:12-13*

Re: Contributing templates

[Joe Gresock]

Thanks Pierre!

So, now that you are PMC, what say you? ;)


Virus-free.
www.avast.com

<#m_4722898169785819424_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

On Sat, May 20, 2017 at 11:50 AM, Pierre Villard <
pierre.villard...@gmail.com> wrote:

> Hey Joe,
>
> I believe there is a confluence page with examples [1] that could be a good
> place but you would need write access on the page or to ask someone with
> write privileges. Or you could add your template as a comment on this page
> I guess. I'll let PMC say what is best ;-)
>
> [1]
> https://cwiki.apache.org/confluence/display/NIFI/
> Example+Dataflow+Templates
>
> Pierre
>
> 2017-05-20 16:55 GMT+02:00 Joe Gresock :
>
> > All,
> >
> > Is there a place to contribute templates?  I have a template for using
> the
> > SiteToSiteStatusReportingTask along with ConvertRecord and
> > PutElasticsearchHttp that automatically stores NiFi's status history in
> an
> > Elasticsearch database for long term analysis of NiFi status history.
> >
> > If there is a place for contributions like this, what is the best way to
> > upload the template/description?
> >
> > --
> > I know what it is to be in need, and I know what it is to have plenty.  I
> > have learned the secret of being content in any and every situation,
> > whether well fed or hungry, whether living in plenty or in want.  I can
> do
> > all this through him who gives me strength.*-Philippians 4:12-13*
> >
>



-- 
I know what it is to be in need, and I know what it is to have plenty.  I
have learned the secret of being content in any and every situation,
whether well fed or hungry, whether living in plenty or in want.  I can do
all this through him who gives me strength.*-Philippians 4:12-13*

Re: [ANNOUNCE] New Apache NiFi PMC Member - Pierre Villard

[Andre]

Pierre,

Your contribution to NiFi takes many forms but I would like to thank you
for the amazing blogs you have put together!

I am sure many in the user and dev lists will vouch that your blog contains
some of the first posts people will refer to when they start to use NiFi
and when they try to step up the ladder on their use of the functionality
of the platform.

Thank you and welcome aboard!!!


On Tue, May 23, 2017 at 5:17 AM, Tony Kurc  wrote:

> On behalf of the Apache NiFi PMC, I am pleased to announce that Pierre
> Villard has accepted the PMC's invitation to join the Apache NiFi PMC.  We
> greatly appreciate all of Pierre's hard work and generous contributions to
> the project. We look forward to continued involvement in the project.
>
> Pierre has been a long time code contributor to both NiFi and MiNiFi. In
> addition to that, he's spurred the community through his blogging -
> teaching people how to use, operate, and monitor NiFi. He's a prolific
> reviewer of contributions, and active in the important task of verifying
> releases.
>
> Please join us in congratulating and welcoming Pierre to the Apache NiFi
> PMC.
>
> Congratulations and welcome, Pierre!
>
> -- Tony
>

Re: [ANNOUNCE] New Apache NiFi PMC Member - Pierre Villard

[Yolanda Davis]

Congrats Pierre!

On Mon, May 22, 2017 at 3:17 PM, Tony Kurc  wrote:

> On behalf of the Apache NiFi PMC, I am pleased to announce that Pierre
> Villard has accepted the PMC's invitation to join the Apache NiFi PMC.  We
> greatly appreciate all of Pierre's hard work and generous contributions to
> the project. We look forward to continued involvement in the project.
>
> Pierre has been a long time code contributor to both NiFi and MiNiFi. In
> addition to that, he's spurred the community through his blogging -
> teaching people how to use, operate, and monitor NiFi. He's a prolific
> reviewer of contributions, and active in the important task of verifying
> releases.
>
> Please join us in congratulating and welcoming Pierre to the Apache NiFi
> PMC.
>
> Congratulations and welcome, Pierre!
>
> -- Tony
>



-- 
--
yolanda.m.da...@gmail.com
@YolandaMDavis

Re: [ANNOUNCE] New Apache NiFi PMC Member - Pierre Villard

[Joe Skora]

Congrats Pierre and welcome.  Thanks for all your work and the blogs!

On Mon, May 22, 2017 at 3:18 PM, Joe Witt  wrote:

> Pierre - you've done a lot of really great work for the community.
> Thanks, congrats and Welcome!
>
>
>
> On Mon, May 22, 2017 at 3:17 PM, Tony Kurc  wrote:
> > On behalf of the Apache NiFi PMC, I am pleased to announce that Pierre
> > Villard has accepted the PMC's invitation to join the Apache NiFi PMC.
> We
> > greatly appreciate all of Pierre's hard work and generous contributions
> to
> > the project. We look forward to continued involvement in the project.
> >
> > Pierre has been a long time code contributor to both NiFi and MiNiFi. In
> > addition to that, he's spurred the community through his blogging -
> > teaching people how to use, operate, and monitor NiFi. He's a prolific
> > reviewer of contributions, and active in the important task of verifying
> > releases.
> >
> > Please join us in congratulating and welcoming Pierre to the Apache NiFi
> > PMC.
> >
> > Congratulations and welcome, Pierre!
> >
> > -- Tony
>

Re: [ANNOUNCE] New Apache NiFi PMC Member - Pierre Villard

[Joe Witt]

Pierre - you've done a lot of really great work for the community.
Thanks, congrats and Welcome!



On Mon, May 22, 2017 at 3:17 PM, Tony Kurc  wrote:
> On behalf of the Apache NiFi PMC, I am pleased to announce that Pierre
> Villard has accepted the PMC's invitation to join the Apache NiFi PMC.  We
> greatly appreciate all of Pierre's hard work and generous contributions to
> the project. We look forward to continued involvement in the project.
>
> Pierre has been a long time code contributor to both NiFi and MiNiFi. In
> addition to that, he's spurred the community through his blogging -
> teaching people how to use, operate, and monitor NiFi. He's a prolific
> reviewer of contributions, and active in the important task of verifying
> releases.
>
> Please join us in congratulating and welcoming Pierre to the Apache NiFi
> PMC.
>
> Congratulations and welcome, Pierre!
>
> -- Tony

[ANNOUNCE] New Apache NiFi PMC Member - Pierre Villard

[Tony Kurc]

On behalf of the Apache NiFi PMC, I am pleased to announce that Pierre
Villard has accepted the PMC's invitation to join the Apache NiFi PMC.  We
greatly appreciate all of Pierre's hard work and generous contributions to
the project. We look forward to continued involvement in the project.

Pierre has been a long time code contributor to both NiFi and MiNiFi. In
addition to that, he's spurred the community through his blogging -
teaching people how to use, operate, and monitor NiFi. He's a prolific
reviewer of contributions, and active in the important task of verifying
releases.

Please join us in congratulating and welcoming Pierre to the Apache NiFi
PMC.

Congratulations and welcome, Pierre!

-- Tony

Re: Duplicated processors when using nifi processors dependency

[Bryan Bende]

Thanks James.

For the abstract AWS processors, I would go with one of the following...

1) Another module under nifi-aws-bundle like:

nifi-aws-bunde
nifi-aws-service-api
nifi-aws-service-api-nar
nifi-aws-abstract-processors (or some similar name)
nifi-aws-processors
nifi-aws-nar

2) A new module under nifi-nar-bundles/nifi-extension-utils like nifi-aws-utils

Either way, the desired result is to have a JAR that someone can
depend on that only brings in the abstract processors.

We have been putting generic abstract processors into
nifi-nar-bundles/nifi-extension-utils /nifi-processor-utils, but since
this case depends on the AWS SDK, then we would be introducing that
dependency to all the NARs that depend on nifi-processor-utils, which
is why I was suggesting a new module.

Let me know if this doesn't make sense.


On Mon, May 22, 2017 at 1:24 AM, James Wing  wrote:
> I created a JIRA for the nifi-aws-bundle structure,
> https://issues.apache.org/jira/browse/NIFI-3950.
>
> Is there a best practice for the abstract classes like Robert describes?
> Are they a better fit for the service API bundle or the processors bundle?
> Or does that become irrelevant once the NAR dependencies are aligned and
> they can be references as Java libraries?
>
> Thanks,
>
> James
>
> On Sun, May 21, 2017 at 10:28 AM, Bryan Bende  wrote:
>
>> Currently the AWS NAR contains the processors, controller service API, and
>> controller service implementation, all in one NAR.
>>
>> Typically the controller service API should be in its own NAR, so there
>> should be something like:
>>
>> nifi-aws-bunde
>> nifi-aws-service-api
>> nifi-aws-service-api-nar
>> nifi-aws-processors
>> nifi-aws-nar
>>
>> The nifi-aws-service-api module would have the
>> AWSCredentialsProviderService interface. The nifi-aws-nar would have a NAR
>> dependency on nifi-aws-service-api-nar.
>>
>> Setting it up this way, you nifi-aws-utils project could have a provided
>> dependency on the nifi-aws-service-api, and then your custom NAR could have
>> a NAR dependency on nifi-aws-service-api-nar.
>>
>> There is some more information here about setting up controller services:
>>
>> https://cwiki.apache.org/confluence/display/NIFI/Maven+
>> Projects+for+Extensions#MavenProjectsforExtensions-
>> LinkingProcessorsandControllerServices > confluence/display/NIFI/Maven+Projects+for+Extensions#
>> MavenProjectsforExtensions-LinkingProcessorsandControllerServices>
>>
>> https://github.com/bbende/nifi-dependency-example <
>> https://github.com/bbende/nifi-dependency-example>
>>
>>
>> > On May 20, 2017, at 10:06 AM, Robert  wrote:
>> >
>> > I tired to do move abstract aws processors from
>> > nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors to newly created
>> > nifi-aws-utils.
>> >
>> > Problem that I face now is with AWSCredentialsProviderControllerService
>> that
>> > imo should stay under nifi-aws-bundle, but it is required by
>> > AbstractAWSCredentialsProviderProcessor that should be moved to
>> > nifi-aws-utils as it is used by abstract aws processors.
>> >
>> > What is the preferred way to solve this?
>> > Should I created new 'nifi-aws-service-bundle' under nifi-nar-bundles and
>> > both nifi-aws-processors and nifi-aws-utils can depend on it?
>> >
>> >
>> >
>> >
>> >
>> > --
>> > View this message in context: http://apache-nifi-developer-
>> list.39713.n7.nabble.com/Duplicated-processors-when-using-nifi-processors-
>> dependency-tp15902p15937.html
>> > Sent from the Apache NiFi Developer List mailing list archive at
>> Nabble.com.
>>
>>