Re: [EXT] Re: [DISCUSS} Closing in on a NiFi 1.4.0 release?

[Jeff]

FYI, I'll be starting the release candidate prep around noon EST on the
22nd, which is less than 24 hours from now.  At that point, I'll be moving
tickets out of 1.4.0.

On Thu, Sep 21, 2017, 3:52 PM Milan Chandna
 wrote:

> Thanks Mark (and Jeff) for clearing this out.
>
> Yes I would like below mentioned Jira (supporting ADLS) and PR to be
> reviewed asap,
> so that we can get it in 1.4.0.
> Jira: https://issues.apache.org/jira/browse/NIFI-4360
> PR: https://github.com/apache/nifi/pull/2158
>
> Though the commits are very stable,
> but would be great to get this reviewed by NIFI experts.
> Really want to see ADLS shining out in NIFI along with its co-offerings
> like HDFS.
>
> P.S: These processors are similar to HDFS (in implementation), if that
> helps for reviewers to take it up.
>
> Regards,
> -Milan.
>
> -Original Message-
> From: Mark Payne [mailto:marka...@hotmail.com]
> Sent: Thursday, September 21, 2017 6:16 PM
> To: dev@nifi.apache.org
> Subject: Re: [EXT] Re: [DISCUSS} Closing in on a NiFi 1.4.0 release?
>
> Milan,
>
> There is no specific deadline. The PR would just need to be merged to
> master by the time that the branch is created for the Release Candidate. I
> don't know exactly when that is going to happen, since I am not the Release
> Manager for this particular release. However, I do expect it to be quite
> soon. If you are able to submit a PR that you'd like to have considered for
> inclusion in the release, you can submit it and let us know so that someone
> will review it if time permits.
>
> That said, if there are issues that need to be addressed before merging
> the PR, I don't think we would hold up the release to get it in unless it
> is considered a "BLOCKER" jira.
>
> Does that help?
>
> Thanks
> -Mark
>
>
> > On Sep 21, 2017, at 4:18 AM, Milan Chandna 
> > 
> wrote:
> >
> > Do we have a deadline to get the issue resolved, to include it in 1.4.0
> release.
> >
> > -Original Message-
> > From: Adam Taft [mailto:a...@adamtaft.com]
> > Sent: Thursday, September 21, 2017 9:20 AM
> > To: dev@nifi.apache.org
> > Subject: Re: [EXT] Re: [DISCUSS} Closing in on a NiFi 1.4.0 release?
> >
> > Here's another good link to try, maybe a little easier to read:
> >
> > https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissue
> > s.apache.org%2Fjira%2Fprojects%2FNIFI%2Fversions%2F12340589=02%7C
> > 01%7CMilan.Chandna%40microsoft.com%7C21581ed9a39548be836e08d500a3e900%
> > 7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636415626344144936=
> > kO%2ByQJA%2FchMrvYYMpUZ%2FGOj%2FYVNEWgbmtahAr6ITrVk%3D=0
> >
> >
> >
> > On Wed, Sep 20, 2017 at 8:01 AM, Brandon DeVries  wrote:
> >
> >> Mayank,
> >>
> >> Try this:
> >>
> >> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissu
> >> e
> >> s.apache.org%2Fjira%2Fissues%2F%3Fjql%3Dproject%2520%25=02%7C01%
> >> 7
> >> CMilan.Chandna%40microsoft.com%7C21581ed9a39548be836e08d500a3e900%7C7
> >> 2
> >> f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636415626344144936=U5d
> >> I
> >> A%2FD0S3yLy4trFBag8IZ3mVJtCD1tWtdLc2hZ2rQ%3D=0
> >> 3D%20NIFI%20AND%20fixVersion%20%3D%201.4.0%20ORDER%20BY%20status%20DE
> >> S
> >> C
> >>
> >> Brandon
> >>
> >>
> >> On Wed, Sep 20, 2017 at 9:57 AM mayank rathi 
> >> wrote:
> >>
> >>> Hello All,
> >>>
> >>> How can we find out list of fixes that will go in 1.4.0 release?
> >>>
> >>> Thanks!!
> >>>
> >>> On Wed, Sep 20, 2017 at 9:53 AM, Brandon DeVries  wrote:
> >>>
>  All,
> 
>  I think we should plan on calling for a vote on Friday.  That gives
>  two days to wrap up any outstanding tickets that anyone feels
>  really belong
> >>> in
>  1.4.  At that point the remaining tickets can be shifted to a
>  future release.
> 
>  If there are tickets that are not getting the attention they need
>  to
> >> make
>  it into the release, let the list know.
> 
>  Any objections?
> 
>  Brandon
> 
>  On Wed, Sep 20, 2017 at 12:32 AM Koji Kawamura
>   >>>
>  wrote:
> 
> > Hi Paul,
> >
> > I was able to reproduce the GenerateTableFetch processor issue
> > reported by NIFI-4395.
> > Please go ahead and provide a PR, I can review it.
> >
> > Thanks,
> > Koji
> >
> > On Wed, Sep 20, 2017 at 1:10 PM, Paul Gibeault (pagibeault)
> >  wrote:
> >> We have submitted this JIRA ticket:
> >> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F
> >> %2Fissues.apache.org%2Fjira%2Fbrowse%2FNIFI-4395=02%7C01%
> >> 7CMilan.Chandna%40microsoft.com%7C21581ed9a39548be836e08d500a3
> >> e900%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636415626344
> >> 144936=cSKg20gPxzXXabttEoU4poDyu8k%2BkE2bFybrIc3NKSI%3D&
> >> reserved=0
> >>
> >> This issue causes GenerateTableFetch processor to 

RE: [EXT] Re: [DISCUSS} Closing in on a NiFi 1.4.0 release?

[Milan Chandna]

Thanks Mark (and Jeff) for clearing this out.

Yes I would like below mentioned Jira (supporting ADLS) and PR to be reviewed 
asap, 
so that we can get it in 1.4.0.
Jira: https://issues.apache.org/jira/browse/NIFI-4360 
PR: https://github.com/apache/nifi/pull/2158 

Though the commits are very stable, 
but would be great to get this reviewed by NIFI experts.
Really want to see ADLS shining out in NIFI along with its co-offerings like 
HDFS.

P.S: These processors are similar to HDFS (in implementation), if that helps 
for reviewers to take it up.

Regards,
-Milan.

-Original Message-
From: Mark Payne [mailto:marka...@hotmail.com] 
Sent: Thursday, September 21, 2017 6:16 PM
To: dev@nifi.apache.org
Subject: Re: [EXT] Re: [DISCUSS} Closing in on a NiFi 1.4.0 release?

Milan,

There is no specific deadline. The PR would just need to be merged to master by 
the time that the branch is created for the Release Candidate. I don't know 
exactly when that is going to happen, since I am not the Release Manager for 
this particular release. However, I do expect it to be quite soon. If you are 
able to submit a PR that you'd like to have considered for inclusion in the 
release, you can submit it and let us know so that someone will review it if 
time permits.

That said, if there are issues that need to be addressed before merging the PR, 
I don't think we would hold up the release to get it in unless it is considered 
a "BLOCKER" jira.

Does that help?

Thanks
-Mark


> On Sep 21, 2017, at 4:18 AM, Milan Chandna 
>  wrote:
> 
> Do we have a deadline to get the issue resolved, to include it in 1.4.0 
> release.
> 
> -Original Message-
> From: Adam Taft [mailto:a...@adamtaft.com]
> Sent: Thursday, September 21, 2017 9:20 AM
> To: dev@nifi.apache.org
> Subject: Re: [EXT] Re: [DISCUSS} Closing in on a NiFi 1.4.0 release?
> 
> Here's another good link to try, maybe a little easier to read:
> 
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissue
> s.apache.org%2Fjira%2Fprojects%2FNIFI%2Fversions%2F12340589=02%7C
> 01%7CMilan.Chandna%40microsoft.com%7C21581ed9a39548be836e08d500a3e900%
> 7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636415626344144936=
> kO%2ByQJA%2FchMrvYYMpUZ%2FGOj%2FYVNEWgbmtahAr6ITrVk%3D=0
> 
> 
> 
> On Wed, Sep 20, 2017 at 8:01 AM, Brandon DeVries  wrote:
> 
>> Mayank,
>> 
>> Try this:
>> 
>> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissu
>> e
>> s.apache.org%2Fjira%2Fissues%2F%3Fjql%3Dproject%2520%25=02%7C01%
>> 7
>> CMilan.Chandna%40microsoft.com%7C21581ed9a39548be836e08d500a3e900%7C7
>> 2 
>> f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636415626344144936=U5d
>> I
>> A%2FD0S3yLy4trFBag8IZ3mVJtCD1tWtdLc2hZ2rQ%3D=0
>> 3D%20NIFI%20AND%20fixVersion%20%3D%201.4.0%20ORDER%20BY%20status%20DE
>> S
>> C
>> 
>> Brandon
>> 
>> 
>> On Wed, Sep 20, 2017 at 9:57 AM mayank rathi 
>> wrote:
>> 
>>> Hello All,
>>> 
>>> How can we find out list of fixes that will go in 1.4.0 release?
>>> 
>>> Thanks!!
>>> 
>>> On Wed, Sep 20, 2017 at 9:53 AM, Brandon DeVries  wrote:
>>> 
 All,
 
 I think we should plan on calling for a vote on Friday.  That gives 
 two days to wrap up any outstanding tickets that anyone feels 
 really belong
>>> in
 1.4.  At that point the remaining tickets can be shifted to a 
 future release.
 
 If there are tickets that are not getting the attention they need 
 to
>> make
 it into the release, let the list know.
 
 Any objections?
 
 Brandon
 
 On Wed, Sep 20, 2017 at 12:32 AM Koji Kawamura 
 >> 
 wrote:
 
> Hi Paul,
> 
> I was able to reproduce the GenerateTableFetch processor issue 
> reported by NIFI-4395.
> Please go ahead and provide a PR, I can review it.
> 
> Thanks,
> Koji
> 
> On Wed, Sep 20, 2017 at 1:10 PM, Paul Gibeault (pagibeault) 
>  wrote:
>> We have submitted this JIRA ticket:
>> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F
>> %2Fissues.apache.org%2Fjira%2Fbrowse%2FNIFI-4395=02%7C01%
>> 7CMilan.Chandna%40microsoft.com%7C21581ed9a39548be836e08d500a3
>> e900%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636415626344
>> 144936=cSKg20gPxzXXabttEoU4poDyu8k%2BkE2bFybrIc3NKSI%3D&
>> reserved=0
>> 
>> This issue causes GenerateTableFetch processor to malfunction
>> after a
> server restart.
>> 
>> We are very interested in getting this released in 1.4.0 and are
 willing
> to provide the PR if there is still time.
>> 
>> Thanks,
>> Paul Gibeault
>> 
>> 
>> -Original Message-
>> From: Michael Hogue [mailto:michael.p.hogu...@gmail.com]
>> Sent: Tuesday, September 19, 2017 9:36 AM
>> To: dev@nifi.apache.org
>> Subject: [EXT] Re: [DISCUSS} Closing in on a NiFi 1.4.0 

Re: Dockerfile and Docker Hub Management

[Adam Taft]

Aldrin,

+1 to separate repository (bullet #2).  The basic premise that Docker
releases should happen separate from the main distribution is spot on. I
think a separate repository would help keep this separation.

I tend to believe that the future of NiFi distributions will be via
containerization. Making the Docker components somewhat a standalone
initiative will help drive changes and innovation in this area.  I'd like
to help see Docker become a first-class citizen for distributing, running
and upgrading NiFi.

Thanks,

Adam



On Thu, Sep 21, 2017 at 9:11 AM, Aldrin Piri  wrote:

> Hey folks,
>
> ** This message turned out to be more detailed than anticipated.  In
> summary, I propose consolidating Docker/container work with a separate
> release process outside of the repository they are packaging.  Full
> thoughts and background follow.  Any input would be appreciated!
>
> ---
>
> I've been working through providing some additional Docker capabilities for
> the project and wanted to share some thoughts as well as possible plans to
> help us be a bit more nimble and responsive to curating Dockerfiles and
> their respective images on DockerHub.
>
> As a bit of context, we currently have the core NiFi project captured in
> two Dockerfiles, one that is used in conjunction with a Maven plugin for
> creating an image during the NiFi build (dockermaven), and another that is
> used for building tagged releases on Docker Hub (dockerhub).  Both of these
> artifacts, currently, reside in a nifi-docker project and are activated via
> Maven profile, (-P docker).
>
> We've seen at times that this is a very coupled process and limits our
> flexibility.  For instance, we had an ill-placed 'chown' which caused a
> duplicating layer and causes our image to be doubly large.  While this has
> been remedied, given current release processes, this is included with the
> core nifi release and we have been unable to rectify that issue.
>
> Another issue is a very specific sequence of actions that needs to happen
> with the current release for artifacts to be triggered correctly in Docker
> Hub.  This can be seen in Section 6 of the release guide [1].  While there
> are ways to rectify this if the timing isn't quite right and/or an error is
> made, it can impose an additional burden on the INFRA team to facilitate
> these requests as there currently is no capability for PMCs to manage their
> Docker repositories directly.
>
> Ultimately, I think we should consider a separate release process for NiFi
> Docker, and any associated efforts that may relate to those files.  In this
> context, NiFi is encompassing of all projects/efforts in the project.
> Additional efforts could comprise of examples of configuring NiFi to be
> secured or clustered, receive data from MiNiFi instances, or using Docker
> Compose or other orchestration frameworks. I have also noticed a number of
> different areas across our work that are using Docker for integration
> testing purposes.  With some planning and coordination, we could likely
> consolidate many of these core resources/templates to allow us to reuse
> them across efforts.
>
> I believe there are two approaches from an organizational standpoint that
> let us execute on the separate release process effectively:
>
> 1.) Condense all Docker artifacts into the current NiFi repository [2].  We
> update our release for NiFi to exclude the Docker subtree to carry out our
> normal release flow and provide the build/tooling for the Docker subtree to
> be released on its own.
>
> 2.)  Establish a new git repository to handle Docker and any other
> containerization efforts and migrate all existing resources into a file
> structure that makes sense.
>
> My inclination is toward (2).
>
> Regardless of path chosen above, this frees us to handle updates and
> improvements to container efforts when needed.  Any time we wanted to
> release updates to Docker images, we could perform a separate release on
> either the subtree of (1) or the repository of (2) and reference the
> associated latest artifacts of NiFi.
>
> If you've made it this far, thanks for working through the wall of text and
> would appreciate any thoughts or comments.
>
> [1] http://nifi.apache.org/release-guide.html
> [2] https://git-wip-us.apache.org/repos/asf?p=nifi.git
>

Re: Dockerfile and Docker Hub Management

[Kevin Doran]

Thanks, Aldrin.

I'm a +1 for a separate NiFi Docker project and release process, allowing 
improvements to the containerization of NiFi artifacts to be made independently 
from the artifacts (and their source files). I have observed many community 
members maintaining their own NiFi Dockerfiles and Docker Compose files, and it 
would be nice if the most reusable variant(s) of these could be well maintained 
in one location on an independent schedule.

I also agree a separate repository would be cleaner and preferable that a 
subtree of the existing NiFi repository.

Thanks,
Kevin

On 9/21/17, 11:12, "Aldrin Piri"  wrote:

Hey folks,

** This message turned out to be more detailed than anticipated.  In
summary, I propose consolidating Docker/container work with a separate
release process outside of the repository they are packaging.  Full
thoughts and background follow.  Any input would be appreciated!

---

I've been working through providing some additional Docker capabilities for
the project and wanted to share some thoughts as well as possible plans to
help us be a bit more nimble and responsive to curating Dockerfiles and
their respective images on DockerHub.

As a bit of context, we currently have the core NiFi project captured in
two Dockerfiles, one that is used in conjunction with a Maven plugin for
creating an image during the NiFi build (dockermaven), and another that is
used for building tagged releases on Docker Hub (dockerhub).  Both of these
artifacts, currently, reside in a nifi-docker project and are activated via
Maven profile, (-P docker).

We've seen at times that this is a very coupled process and limits our
flexibility.  For instance, we had an ill-placed 'chown' which caused a
duplicating layer and causes our image to be doubly large.  While this has
been remedied, given current release processes, this is included with the
core nifi release and we have been unable to rectify that issue.

Another issue is a very specific sequence of actions that needs to happen
with the current release for artifacts to be triggered correctly in Docker
Hub.  This can be seen in Section 6 of the release guide [1].  While there
are ways to rectify this if the timing isn't quite right and/or an error is
made, it can impose an additional burden on the INFRA team to facilitate
these requests as there currently is no capability for PMCs to manage their
Docker repositories directly.

Ultimately, I think we should consider a separate release process for NiFi
Docker, and any associated efforts that may relate to those files.  In this
context, NiFi is encompassing of all projects/efforts in the project.
Additional efforts could comprise of examples of configuring NiFi to be
secured or clustered, receive data from MiNiFi instances, or using Docker
Compose or other orchestration frameworks. I have also noticed a number of
different areas across our work that are using Docker for integration
testing purposes.  With some planning and coordination, we could likely
consolidate many of these core resources/templates to allow us to reuse
them across efforts.

I believe there are two approaches from an organizational standpoint that
let us execute on the separate release process effectively:

1.) Condense all Docker artifacts into the current NiFi repository [2].  We
update our release for NiFi to exclude the Docker subtree to carry out our
normal release flow and provide the build/tooling for the Docker subtree to
be released on its own.

2.)  Establish a new git repository to handle Docker and any other
containerization efforts and migrate all existing resources into a file
structure that makes sense.

My inclination is toward (2).

Regardless of path chosen above, this frees us to handle updates and
improvements to container efforts when needed.  Any time we wanted to
release updates to Docker images, we could perform a separate release on
either the subtree of (1) or the repository of (2) and reference the
associated latest artifacts of NiFi.

If you've made it this far, thanks for working through the wall of text and
would appreciate any thoughts or comments.

[1] http://nifi.apache.org/release-guide.html
[2] https://git-wip-us.apache.org/repos/asf?p=nifi.git

about putHiveStreaming processor

[????]

hello:
I'm trying to write the Hive using NiFi's PutHiveStreaming processor. I 
faced this  issue with Nifi 1.3.0 and HDP-2.6, myhive table is partioned , 
bucketed and stored as ORC format.
Any input would be greatly appreciated.  I have validated the network 
configuration is correct.


First,In the thrift metastore log,it shows like :



Second,in the nifi-app.log,it shows like:




   Thank very much.
   Best wishes