Re: Status of nifi-docker

[Aldrin Piri]

We may be lacking in that department but it is via a profile. Give -Pdocker as 
part of your maven build and it should build off your current source. 

> On Jun 29, 2018, at 19:32, Mike Thomsen  wrote:
> 
> Is there any documentation on how to build the dockermaven one? It just
> generates a 7kb jar for me.
> 
> Thanks,
> 
> Mike
> 
>> On Fri, Jun 29, 2018 at 10:36 AM Jeremy Dyer  wrote:
>> 
>> Mike - Yes both of them are used. The dockerhub one is obviously pushed to
>> dockerhub (and built there hence the separate file) and the dockermaven one
>> is built locally for anyone who wishes to do so. I know I use the
>> dockermaven one all the time for development purposes
>> 
>> Thanks,
>> Jeremy Dyer
>> 
>> Thanks - Jeremy Dyer
>> 
>> From: Pierre Villard 
>> Sent: Friday, June 29, 2018 10:11:31 AM
>> To: dev
>> Subject: Re: Status of nifi-docker
>> 
>> Wrong copy/paste... thread with title Question on Docker module
>> <
>> https://mail-archives.apache.org/mod_mbox/nifi-dev/201806.mbox/ajax/%3CCAGvqHsdf5fNuC2mGwmGWZRjQ3_T5W_MqtyB9KY9k7JGpEaL8yQ%40mail.gmail.com%3E
>>> 
>> 
>> 
>> 2018-06-29 16:10 GMT+02:00 Pierre Villard :
>> 
>>> Hi Mike,
>>> 
>>> Have a look on this thread: https://mail-archives.apache.
>>> org/mod_mbox/nifi-dev/201806.mbox/browser
>>> 
>>> Pierre
>>> 
>>> 2018-06-29 15:59 GMT+02:00 Mike Thomsen :
>>> 
 We appear to have two copies of the Docker builds. One in dockerhub and
 one
 in dockermaven. Do we use both of them? I need to know so I can try to
 finish up a PR to expose ZK-based clustering w/ Docker.
 
 Thanks,
 
 Mike
 
>>> 
>>> 
>> 

Re: Status of nifi-docker

[Mike Thomsen]

Is there any documentation on how to build the dockermaven one? It just
generates a 7kb jar for me.

Thanks,

Mike

On Fri, Jun 29, 2018 at 10:36 AM Jeremy Dyer  wrote:

> Mike - Yes both of them are used. The dockerhub one is obviously pushed to
> dockerhub (and built there hence the separate file) and the dockermaven one
> is built locally for anyone who wishes to do so. I know I use the
> dockermaven one all the time for development purposes
>
> Thanks,
> Jeremy Dyer
>
> Thanks - Jeremy Dyer
> 
> From: Pierre Villard 
> Sent: Friday, June 29, 2018 10:11:31 AM
> To: dev
> Subject: Re: Status of nifi-docker
>
> Wrong copy/paste... thread with title Question on Docker module
> <
> https://mail-archives.apache.org/mod_mbox/nifi-dev/201806.mbox/ajax/%3CCAGvqHsdf5fNuC2mGwmGWZRjQ3_T5W_MqtyB9KY9k7JGpEaL8yQ%40mail.gmail.com%3E
> >
>
>
> 2018-06-29 16:10 GMT+02:00 Pierre Villard :
>
> > Hi Mike,
> >
> > Have a look on this thread: https://mail-archives.apache.
> > org/mod_mbox/nifi-dev/201806.mbox/browser
> >
> > Pierre
> >
> > 2018-06-29 15:59 GMT+02:00 Mike Thomsen :
> >
> >> We appear to have two copies of the Docker builds. One in dockerhub and
> >> one
> >> in dockermaven. Do we use both of them? I need to know so I can try to
> >> finish up a PR to expose ZK-based clustering w/ Docker.
> >>
> >> Thanks,
> >>
> >> Mike
> >>
> >
> >
>

Re: [VOTE] Release Apache NiFi MiNiFi 0.5.0 RC2

[Andy LoPresto]

+1, binding

Verified all keys and checksums, ran through all the builds, verified tests and 
contrib-check, and the existence of LICENSE/NOTICE/README at appropriate 
locations. A couple minor points on the release process below (and some other 
non-blocking issues that I will open as Jiras).

* Now that MD5 is deprecated, I believe we agreed to provide a SHA-512 
signature as well. The release guide should be updated to include this for next 
time.
* The release helper should specify that LICENSE, NOTICE, and README are in 
minifi-assembly/target/minifi-0.5.0-bin/minifi-0.5.0/

Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Jun 29, 2018, at 8:29 AM, Aldrin Piri  wrote:
> 
> Hi folks,
> 
> I worked with Jeremy to resolve the discrepancy and we were able to get the
> commits/tag in question from the Maven release plugin pushed to the repo.
> 
> The minifi-0.5.0-RC2 tag is correct in pointing to the source of the...
> source commit 58b8c598c0866c8f1200164ab14f3df0d632d522 which was initiated
> from the last commit on master, 05f516d3da33a0547ccfad342414504cdacd4a68.
> 
> Links for reference:
> tag:
> * https://github.com/apache/nifi-minifi/tree/minifi-0.5.0-RC2
> *
> https://git-wip-us.apache.org/repos/asf?p=nifi-minifi.git;a=shortlog;h=refs/tags/minifi-0.5.0-RC2
> 
> RC2 branch commit:
> *
> https://github.com/apiri/nifi-minifi/commit/58b8c598c0866c8f1200164ab14f3df0d632d522
> *
> https://git-wip-us.apache.org/repos/asf?p=nifi-minifi.git;a=commit;h=58b8c598c0866c8f1200164ab14f3df0d632d522
> 
> --aldrin
> 
> 
> On Fri, Jun 29, 2018 at 7:59 AM Jeff Zemerick  wrote:
> 
>> +1 non-binding
>> 
>> Built and ran MiNiFi and C2 without issues.
>> 
>> On Thu, Jun 28, 2018 at 3:59 PM Aldrin Piri  wrote:
>> 
>>> +1, binding
>>> 
>>> comments:
>>> My vote is on the assumption there there was a slight mix up with tagging
>>> and it can be easily remedied.  The
>>> 05f516d3da33a0547ccfad342414504cdacd4a68
>>> points to the last commit and a snapshot version.  This is correct from
>>> where the release is started but does not capture the generated prepared
>>> release artifacts.  I believe if that is pushed and referenced, we will
>>> have what is packed in the source distribution, which is listed as 0.5.0,
>>> non-snapshot.
>>> 
>>> signature and hashes looked good
>>> verified contrib, build, and tests of minifi, config-toolkit, and c2
>> server
>>> verified transform functionality of config toolkit
>>> verified consumption of transformed config in minifi and its successful
>>> site to site transaction to nifi
>>> verified interaction with c2 server with a file based provider
>>> 
>>> On Thu, Jun 28, 2018 at 1:19 PM Jeremy Dyer 
>> wrote:
>>> 
 Hello,
 
 I am pleased to call this vote for the source release of Apache NiFi
>>> MiNiFi
 minifi-0.5.0 RC2.
 
 The source zip, including signatures, digests, etc. can be found at:
 *
>> https://repository.apache.org/content/repositories/orgapachenifi-1130/
 <
>> https://repository.apache.org/content/repositories/orgapachenifi-1130/
 *
 
 The Git tag is minifi-0.5.0-RC2
 The Git commit ID is 05f516d3da33a0547ccfad342414504cdacd4a68
 *
 
>>> 
>> https://git-wip-us.apache.org/repos/asf?p=nifi-minifi.git;a=commit;h=05f516d3da33a0547ccfad342414504cdacd4a68
 <
 
>>> 
>> https://git-wip-us.apache.org/repos/asf?p=nifi-minifi.git;a=commit;h=05f516d3da33a0547ccfad342414504cdacd4a68
> *
 *
 
>>> 
>> https://github.com/apache/nifi-minifi/commit/05f516d3da33a0547ccfad342414504cdacd4a68
 <
 
>>> 
>> https://github.com/apache/nifi-minifi/commit/05f516d3da33a0547ccfad342414504cdacd4a68
> *
 
 wget
 
 
>>> 
>> https://repository.apache.org/content/repositories/orgapachenifi-1130/org/apache/nifi/minifi/minifi/0.5.0/minifi-0.5.0-source-release.zip
 
 Checksums of minifi-0.5.0-source-release.zip:
 SHA1: be624db030aedd5c249e58c4c57d55ca917cd6ea
 SHA256:
>> 1f96ca7d8c2f52f9c15dad532065163f14cb01a2edbc783d4faa33656ff5ab88
 
 Release artifacts are signed with the following key:
 https://people.apache.org/keys/committer/jeremydyer.asc
 
 KEYS file available here:
 https://dist.apache.org/repos/dist/release/nifi/KEYS
 
 16 issues were closed/resolved for this release:
 *
 
>>> 
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319921=12342658
 <
 
>>> 
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319921=12342658
> *
 
 Release note highlights can be found here:
 https://cwiki.apache.org/confluence/display/MINIFI/Release+
 Notes#ReleaseNotes-Version0.5.0
 
 The vote will be open until 2:00PM EDT, 1 July 2018.
 
 Please download the release candidate and evaluate the necessary items
 including checking hashes, signatures, build
 from source, and test.  Then please 

Re: [VOTE] Release Apache NiFi MiNiFi 0.5.0 RC2

[Aldrin Piri]

Hi folks,

I worked with Jeremy to resolve the discrepancy and we were able to get the
commits/tag in question from the Maven release plugin pushed to the repo.

The minifi-0.5.0-RC2 tag is correct in pointing to the source of the...
source commit 58b8c598c0866c8f1200164ab14f3df0d632d522 which was initiated
from the last commit on master, 05f516d3da33a0547ccfad342414504cdacd4a68.

Links for reference:
tag:
* https://github.com/apache/nifi-minifi/tree/minifi-0.5.0-RC2
*
https://git-wip-us.apache.org/repos/asf?p=nifi-minifi.git;a=shortlog;h=refs/tags/minifi-0.5.0-RC2

RC2 branch commit:
*
https://github.com/apiri/nifi-minifi/commit/58b8c598c0866c8f1200164ab14f3df0d632d522
*
https://git-wip-us.apache.org/repos/asf?p=nifi-minifi.git;a=commit;h=58b8c598c0866c8f1200164ab14f3df0d632d522

--aldrin


On Fri, Jun 29, 2018 at 7:59 AM Jeff Zemerick  wrote:

> +1 non-binding
>
> Built and ran MiNiFi and C2 without issues.
>
> On Thu, Jun 28, 2018 at 3:59 PM Aldrin Piri  wrote:
>
> > +1, binding
> >
> > comments:
> > My vote is on the assumption there there was a slight mix up with tagging
> > and it can be easily remedied.  The
> > 05f516d3da33a0547ccfad342414504cdacd4a68
> > points to the last commit and a snapshot version.  This is correct from
> > where the release is started but does not capture the generated prepared
> > release artifacts.  I believe if that is pushed and referenced, we will
> > have what is packed in the source distribution, which is listed as 0.5.0,
> > non-snapshot.
> >
> > signature and hashes looked good
> > verified contrib, build, and tests of minifi, config-toolkit, and c2
> server
> > verified transform functionality of config toolkit
> > verified consumption of transformed config in minifi and its successful
> > site to site transaction to nifi
> > verified interaction with c2 server with a file based provider
> >
> > On Thu, Jun 28, 2018 at 1:19 PM Jeremy Dyer 
> wrote:
> >
> > > Hello,
> > >
> > > I am pleased to call this vote for the source release of Apache NiFi
> > MiNiFi
> > > minifi-0.5.0 RC2.
> > >
> > > The source zip, including signatures, digests, etc. can be found at:
> > > *
> https://repository.apache.org/content/repositories/orgapachenifi-1130/
> > > <
> https://repository.apache.org/content/repositories/orgapachenifi-1130/
> > >*
> > >
> > > The Git tag is minifi-0.5.0-RC2
> > > The Git commit ID is 05f516d3da33a0547ccfad342414504cdacd4a68
> > > *
> > >
> >
> https://git-wip-us.apache.org/repos/asf?p=nifi-minifi.git;a=commit;h=05f516d3da33a0547ccfad342414504cdacd4a68
> > > <
> > >
> >
> https://git-wip-us.apache.org/repos/asf?p=nifi-minifi.git;a=commit;h=05f516d3da33a0547ccfad342414504cdacd4a68
> > > >*
> > > *
> > >
> >
> https://github.com/apache/nifi-minifi/commit/05f516d3da33a0547ccfad342414504cdacd4a68
> > > <
> > >
> >
> https://github.com/apache/nifi-minifi/commit/05f516d3da33a0547ccfad342414504cdacd4a68
> > > >*
> > >
> > > wget
> > >
> > >
> >
> https://repository.apache.org/content/repositories/orgapachenifi-1130/org/apache/nifi/minifi/minifi/0.5.0/minifi-0.5.0-source-release.zip
> > >
> > > Checksums of minifi-0.5.0-source-release.zip:
> > > SHA1: be624db030aedd5c249e58c4c57d55ca917cd6ea
> > > SHA256:
> 1f96ca7d8c2f52f9c15dad532065163f14cb01a2edbc783d4faa33656ff5ab88
> > >
> > > Release artifacts are signed with the following key:
> > > https://people.apache.org/keys/committer/jeremydyer.asc
> > >
> > > KEYS file available here:
> > > https://dist.apache.org/repos/dist/release/nifi/KEYS
> > >
> > > 16 issues were closed/resolved for this release:
> > > *
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319921=12342658
> > > <
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319921=12342658
> > > >*
> > >
> > > Release note highlights can be found here:
> > > https://cwiki.apache.org/confluence/display/MINIFI/Release+
> > > Notes#ReleaseNotes-Version0.5.0
> > >
> > > The vote will be open until 2:00PM EDT, 1 July 2018.
> > >
> > > Please download the release candidate and evaluate the necessary items
> > > including checking hashes, signatures, build
> > > from source, and test.  Then please vote:
> > >
> > > [ ] +1 Release this package as minifi-0.4.0
> > > [ ] +0 no opinion
> > > [ ] -1 Do not release this package because...
> > >
> > > Thanks!
> > >
> >
>

Re: Status of nifi-docker

[Jeremy Dyer]

Mike - Yes both of them are used. The dockerhub one is obviously pushed to 
dockerhub (and built there hence the separate file) and the dockermaven one is 
built locally for anyone who wishes to do so. I know I use the dockermaven one 
all the time for development purposes

Thanks,
Jeremy Dyer

Thanks - Jeremy Dyer

From: Pierre Villard 
Sent: Friday, June 29, 2018 10:11:31 AM
To: dev
Subject: Re: Status of nifi-docker

Wrong copy/paste... thread with title Question on Docker module



2018-06-29 16:10 GMT+02:00 Pierre Villard :

> Hi Mike,
>
> Have a look on this thread: https://mail-archives.apache.
> org/mod_mbox/nifi-dev/201806.mbox/browser
>
> Pierre
>
> 2018-06-29 15:59 GMT+02:00 Mike Thomsen :
>
>> We appear to have two copies of the Docker builds. One in dockerhub and
>> one
>> in dockermaven. Do we use both of them? I need to know so I can try to
>> finish up a PR to expose ZK-based clustering w/ Docker.
>>
>> Thanks,
>>
>> Mike
>>
>
>

Re: Need to be added to the Registry Jira project

[Aldrin Piri]

Mike,

I added mike.thomsen.  Let me know if it's not working.

On Fri, Jun 29, 2018 at 9:49 AM Mike Thomsen  wrote:

> Can someone set me up as a contributor?
>
> Thanks,
>
> Mike
>

Re: Status of nifi-docker

[Pierre Villard]

Wrong copy/paste... thread with title Question on Docker module



2018-06-29 16:10 GMT+02:00 Pierre Villard :

> Hi Mike,
>
> Have a look on this thread: https://mail-archives.apache.
> org/mod_mbox/nifi-dev/201806.mbox/browser
>
> Pierre
>
> 2018-06-29 15:59 GMT+02:00 Mike Thomsen :
>
>> We appear to have two copies of the Docker builds. One in dockerhub and
>> one
>> in dockermaven. Do we use both of them? I need to know so I can try to
>> finish up a PR to expose ZK-based clustering w/ Docker.
>>
>> Thanks,
>>
>> Mike
>>
>
>

Re: Status of nifi-docker

[Pierre Villard]

Hi Mike,

Have a look on this thread:
https://mail-archives.apache.org/mod_mbox/nifi-dev/201806.mbox/browser

Pierre

2018-06-29 15:59 GMT+02:00 Mike Thomsen :

> We appear to have two copies of the Docker builds. One in dockerhub and one
> in dockermaven. Do we use both of them? I need to know so I can try to
> finish up a PR to expose ZK-based clustering w/ Docker.
>
> Thanks,
>
> Mike
>

Status of nifi-docker

[Mike Thomsen]

We appear to have two copies of the Docker builds. One in dockerhub and one
in dockermaven. Do we use both of them? I need to know so I can try to
finish up a PR to expose ZK-based clustering w/ Docker.

Thanks,

Mike

Need to be added to the Registry Jira project

[Mike Thomsen]

Can someone set me up as a contributor?

Thanks,

Mike

Re: [VOTE] Release Apache NiFi MiNiFi 0.5.0 RC2

[Jeff Zemerick]

+1 non-binding

Built and ran MiNiFi and C2 without issues.

On Thu, Jun 28, 2018 at 3:59 PM Aldrin Piri  wrote:

> +1, binding
>
> comments:
> My vote is on the assumption there there was a slight mix up with tagging
> and it can be easily remedied.  The
> 05f516d3da33a0547ccfad342414504cdacd4a68
> points to the last commit and a snapshot version.  This is correct from
> where the release is started but does not capture the generated prepared
> release artifacts.  I believe if that is pushed and referenced, we will
> have what is packed in the source distribution, which is listed as 0.5.0,
> non-snapshot.
>
> signature and hashes looked good
> verified contrib, build, and tests of minifi, config-toolkit, and c2 server
> verified transform functionality of config toolkit
> verified consumption of transformed config in minifi and its successful
> site to site transaction to nifi
> verified interaction with c2 server with a file based provider
>
> On Thu, Jun 28, 2018 at 1:19 PM Jeremy Dyer  wrote:
>
> > Hello,
> >
> > I am pleased to call this vote for the source release of Apache NiFi
> MiNiFi
> > minifi-0.5.0 RC2.
> >
> > The source zip, including signatures, digests, etc. can be found at:
> > *https://repository.apache.org/content/repositories/orgapachenifi-1130/
> >  >*
> >
> > The Git tag is minifi-0.5.0-RC2
> > The Git commit ID is 05f516d3da33a0547ccfad342414504cdacd4a68
> > *
> >
> https://git-wip-us.apache.org/repos/asf?p=nifi-minifi.git;a=commit;h=05f516d3da33a0547ccfad342414504cdacd4a68
> > <
> >
> https://git-wip-us.apache.org/repos/asf?p=nifi-minifi.git;a=commit;h=05f516d3da33a0547ccfad342414504cdacd4a68
> > >*
> > *
> >
> https://github.com/apache/nifi-minifi/commit/05f516d3da33a0547ccfad342414504cdacd4a68
> > <
> >
> https://github.com/apache/nifi-minifi/commit/05f516d3da33a0547ccfad342414504cdacd4a68
> > >*
> >
> > wget
> >
> >
> https://repository.apache.org/content/repositories/orgapachenifi-1130/org/apache/nifi/minifi/minifi/0.5.0/minifi-0.5.0-source-release.zip
> >
> > Checksums of minifi-0.5.0-source-release.zip:
> > SHA1: be624db030aedd5c249e58c4c57d55ca917cd6ea
> > SHA256: 1f96ca7d8c2f52f9c15dad532065163f14cb01a2edbc783d4faa33656ff5ab88
> >
> > Release artifacts are signed with the following key:
> > https://people.apache.org/keys/committer/jeremydyer.asc
> >
> > KEYS file available here:
> > https://dist.apache.org/repos/dist/release/nifi/KEYS
> >
> > 16 issues were closed/resolved for this release:
> > *
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319921=12342658
> > <
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319921=12342658
> > >*
> >
> > Release note highlights can be found here:
> > https://cwiki.apache.org/confluence/display/MINIFI/Release+
> > Notes#ReleaseNotes-Version0.5.0
> >
> > The vote will be open until 2:00PM EDT, 1 July 2018.
> >
> > Please download the release candidate and evaluate the necessary items
> > including checking hashes, signatures, build
> > from source, and test.  Then please vote:
> >
> > [ ] +1 Release this package as minifi-0.4.0
> > [ ] +0 no opinion
> > [ ] -1 Do not release this package because...
> >
> > Thanks!
> >
>

Re: [DISCUSS] Tar + Gzip vs. Zip

[Peter Wilcsinszky]

Yes, I mean with this (multistage build) we cannot get rid of the two
separate modules (maven and dockerhub) but we can get rid of the ADD
instruction which I think has the benefit of making the build clearer and
more explicit as well.

On Fri, Jun 29, 2018 at 1:23 PM Aldrin Piri  wrote:

> Hi Peter,
>
> I remember seeing this but the criteria about working only on Mac and
> Windows makes it a challenge, in my opinion.
>
> I also need to apologize as I certainly confused the Dockerfiles between
> the Maven plugin and the Docker Hub.  My prior email should have been
> directed toward the Maven scenario as that is using the ADD.  Docker Hub
> will just require an updating of the curl command to the .zip extension and
> we should be set.  Regardless, Andy, when you make the issue for this
> change feel free to create a subtask of that to update the Dockerfiles.
> Looks like Peter is up to the task but I am also happy to help make the
> adjustments and verify.  The first linked item you provided is the
> multistage approach mentioned.  Multistage builds allow you to effectively
> create throw away images only selecting specific artifacts from them to use
> in a new image.
>
> Thanks!
> --aldrin
>
> On Fri, Jun 29, 2018 at 7:11 AM Peter Wilcsinszky <
> peterwilcsins...@gmail.com> wrote:
>
> > Hi,
> >
> > I wrote about a different solution for which I implemented a PoC for in
> >
> >
> https://lists.apache.org/thread.html/6122674030b8f99a63d586dcdbdaf6b31841572aed63fcc9dcfb5eea@%3Cdev.nifi.apache.org%3E
> > but multistage build could be a better option and I'm happy to create an
> > issue and fix it for the next release.
> >
> > On Fri, Jun 29, 2018 at 3:42 AM Andy LoPresto 
> > wrote:
> >
> > > Thanks Aldrin. I am not knowledgeable on Docker — do either of these
> > > options help us? We could also use a RUN to curl the Zip resource and
> > COPY
> > > the unzipped directory?
> > >
> > > [1] https://github.com/moby/moby/issues/15036#issuecomment-322177465
> > > [2] https://github.com/jlhawn/dockramp
> > >
> > >
> > > Andy LoPresto
> > > alopre...@apache.org
> > > *alopresto.apa...@gmail.com *
> > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> > >
> > > On Jun 28, 2018, at 6:22 PM, Aldrin Piri  wrote:
> > >
> > > Be mindful to also update the Dockerfile used for Docker Hub as this
> will
> > > require some adjustments.  Unfortunately, the ADD instruction does not
> > > support zip files.  This isn't a major inconvenience but will require a
> > > multi-stage build to help keep our image size svelte.  I believe we
> > should
> > > be safe as we have been publishing both tarballs and zips for prior
> > > releases, so the Dockerfile should still work in that scenario.
> > >
> > > On Wed, Jun 27, 2018 at 4:06 PM Andy LoPresto 
> > > wrote:
> > >
> > > Thanks for everyone’s input. It seems to be a clear consensus to
> > eliminate
> > > .tar.gz and only provide .zip moving forward. I’d like to keep this
> > > discussion thread going for another day or two to field any objections.
> > > After that time (Friday-ish), I’ll create a Jira to do this unless
> things
> > > change.
> > >
> > > I will probably keep the possibility to generate the .tar.gz through an
> > > inactive profile to allow people who need that offering to use it.
> There
> > > will be a subtask Jira to update the release guide moving forward as
> > well.
> > >
> > >
> > > Andy LoPresto
> > > alopre...@apache.org
> > > *alopresto.apa...@gmail.com *
> > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> > >
> > > On Jun 26, 2018, at 7:52 PM, James Wing  wrote:
> > >
> > > It's a great idea, Andy, I strongly support just one format.  I think
> Zip
> > > is a good choice.
> > >
> > > On Tue, Jun 26, 2018 at 11:16 AM Otto Fowler 
> > > wrote:
> > >
> > > I end up using zip all the time.  zip +1
> > >
> > >
> > > On June 26, 2018 at 13:30:33, Tony Kurc (tk...@apache.org) wrote:
> > >
> > > My preference is zip.
> > >
> > > On Tue, Jun 26, 2018, 9:21 AM Josh Elser  wrote:
> > >
> > >
> > >
> > > On 6/25/18 11:34 PM, Andy LoPresto wrote:
> > >
> > > Hi folks,
> > >
> > > I do not want to start a long-running argument or entrenched battle.
> > > However, having just performed the RM duties for the latest release, I
> > > believe I have identified a resource inefficiency in the fact that we
> > > generate, upload, host, and distribute two compressed archives of the
> > > binary which are functionally equivalent. For 1.7.0, both the .tar.gz
> > > and .zip files are 1.2 GB (1_224_352_000 bytes for tar.gz vs.
> > > 1_224_392_000 bytes for zip). The time to build and sign these is
> > > substantial, but the true cost comes in uploading and hosting them.
> > > While the fabled extension registry will save all of us from this
> > > burden, it isn’t arriving tomorrow, and I think we could drastically
> > > improve this before the next release.
> > >
> > > I have no personal preference between the two formats. In earlier days,

Re: [DISCUSS] Tar + Gzip vs. Zip

[Aldrin Piri]

Hi Peter,

I remember seeing this but the criteria about working only on Mac and
Windows makes it a challenge, in my opinion.

I also need to apologize as I certainly confused the Dockerfiles between
the Maven plugin and the Docker Hub.  My prior email should have been
directed toward the Maven scenario as that is using the ADD.  Docker Hub
will just require an updating of the curl command to the .zip extension and
we should be set.  Regardless, Andy, when you make the issue for this
change feel free to create a subtask of that to update the Dockerfiles.
Looks like Peter is up to the task but I am also happy to help make the
adjustments and verify.  The first linked item you provided is the
multistage approach mentioned.  Multistage builds allow you to effectively
create throw away images only selecting specific artifacts from them to use
in a new image.

Thanks!
--aldrin

On Fri, Jun 29, 2018 at 7:11 AM Peter Wilcsinszky <
peterwilcsins...@gmail.com> wrote:

> Hi,
>
> I wrote about a different solution for which I implemented a PoC for in
>
> https://lists.apache.org/thread.html/6122674030b8f99a63d586dcdbdaf6b31841572aed63fcc9dcfb5eea@%3Cdev.nifi.apache.org%3E
> but multistage build could be a better option and I'm happy to create an
> issue and fix it for the next release.
>
> On Fri, Jun 29, 2018 at 3:42 AM Andy LoPresto 
> wrote:
>
> > Thanks Aldrin. I am not knowledgeable on Docker — do either of these
> > options help us? We could also use a RUN to curl the Zip resource and
> COPY
> > the unzipped directory?
> >
> > [1] https://github.com/moby/moby/issues/15036#issuecomment-322177465
> > [2] https://github.com/jlhawn/dockramp
> >
> >
> > Andy LoPresto
> > alopre...@apache.org
> > *alopresto.apa...@gmail.com *
> > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> >
> > On Jun 28, 2018, at 6:22 PM, Aldrin Piri  wrote:
> >
> > Be mindful to also update the Dockerfile used for Docker Hub as this will
> > require some adjustments.  Unfortunately, the ADD instruction does not
> > support zip files.  This isn't a major inconvenience but will require a
> > multi-stage build to help keep our image size svelte.  I believe we
> should
> > be safe as we have been publishing both tarballs and zips for prior
> > releases, so the Dockerfile should still work in that scenario.
> >
> > On Wed, Jun 27, 2018 at 4:06 PM Andy LoPresto 
> > wrote:
> >
> > Thanks for everyone’s input. It seems to be a clear consensus to
> eliminate
> > .tar.gz and only provide .zip moving forward. I’d like to keep this
> > discussion thread going for another day or two to field any objections.
> > After that time (Friday-ish), I’ll create a Jira to do this unless things
> > change.
> >
> > I will probably keep the possibility to generate the .tar.gz through an
> > inactive profile to allow people who need that offering to use it. There
> > will be a subtask Jira to update the release guide moving forward as
> well.
> >
> >
> > Andy LoPresto
> > alopre...@apache.org
> > *alopresto.apa...@gmail.com *
> > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> >
> > On Jun 26, 2018, at 7:52 PM, James Wing  wrote:
> >
> > It's a great idea, Andy, I strongly support just one format.  I think Zip
> > is a good choice.
> >
> > On Tue, Jun 26, 2018 at 11:16 AM Otto Fowler 
> > wrote:
> >
> > I end up using zip all the time.  zip +1
> >
> >
> > On June 26, 2018 at 13:30:33, Tony Kurc (tk...@apache.org) wrote:
> >
> > My preference is zip.
> >
> > On Tue, Jun 26, 2018, 9:21 AM Josh Elser  wrote:
> >
> >
> >
> > On 6/25/18 11:34 PM, Andy LoPresto wrote:
> >
> > Hi folks,
> >
> > I do not want to start a long-running argument or entrenched battle.
> > However, having just performed the RM duties for the latest release, I
> > believe I have identified a resource inefficiency in the fact that we
> > generate, upload, host, and distribute two compressed archives of the
> > binary which are functionally equivalent. For 1.7.0, both the .tar.gz
> > and .zip files are 1.2 GB (1_224_352_000 bytes for tar.gz vs.
> > 1_224_392_000 bytes for zip). The time to build and sign these is
> > substantial, but the true cost comes in uploading and hosting them.
> > While the fabled extension registry will save all of us from this
> > burden, it isn’t arriving tomorrow, and I think we could drastically
> > improve this before the next release.
> >
> > I have no personal preference between the two formats. In earlier days,
> > there were platform inconsistencies and the tools weren’t available on
> > all systems, but now they are pretty standard for all users. This [1]
> >
> > is
> >
> > an interesting article I found which had some good info on the origins,
> > and here are some additional resources for anyone interested [2][3]. I
> > don’t care which we pick, but I propose removing one of the options for
> > the build going forward (toolkit as well).
> >
> > That said, if someone has a good reason that both are necessary, I
> >
> > 

Re: [DISCUSS] Tar + Gzip vs. Zip

[Peter Wilcsinszky]

Hi,

I wrote about a different solution for which I implemented a PoC for in
https://lists.apache.org/thread.html/6122674030b8f99a63d586dcdbdaf6b31841572aed63fcc9dcfb5eea@%3Cdev.nifi.apache.org%3E
but multistage build could be a better option and I'm happy to create an
issue and fix it for the next release.

On Fri, Jun 29, 2018 at 3:42 AM Andy LoPresto  wrote:

> Thanks Aldrin. I am not knowledgeable on Docker — do either of these
> options help us? We could also use a RUN to curl the Zip resource and COPY
> the unzipped directory?
>
> [1] https://github.com/moby/moby/issues/15036#issuecomment-322177465
> [2] https://github.com/jlhawn/dockramp
>
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Jun 28, 2018, at 6:22 PM, Aldrin Piri  wrote:
>
> Be mindful to also update the Dockerfile used for Docker Hub as this will
> require some adjustments.  Unfortunately, the ADD instruction does not
> support zip files.  This isn't a major inconvenience but will require a
> multi-stage build to help keep our image size svelte.  I believe we should
> be safe as we have been publishing both tarballs and zips for prior
> releases, so the Dockerfile should still work in that scenario.
>
> On Wed, Jun 27, 2018 at 4:06 PM Andy LoPresto 
> wrote:
>
> Thanks for everyone’s input. It seems to be a clear consensus to eliminate
> .tar.gz and only provide .zip moving forward. I’d like to keep this
> discussion thread going for another day or two to field any objections.
> After that time (Friday-ish), I’ll create a Jira to do this unless things
> change.
>
> I will probably keep the possibility to generate the .tar.gz through an
> inactive profile to allow people who need that offering to use it. There
> will be a subtask Jira to update the release guide moving forward as well.
>
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Jun 26, 2018, at 7:52 PM, James Wing  wrote:
>
> It's a great idea, Andy, I strongly support just one format.  I think Zip
> is a good choice.
>
> On Tue, Jun 26, 2018 at 11:16 AM Otto Fowler 
> wrote:
>
> I end up using zip all the time.  zip +1
>
>
> On June 26, 2018 at 13:30:33, Tony Kurc (tk...@apache.org) wrote:
>
> My preference is zip.
>
> On Tue, Jun 26, 2018, 9:21 AM Josh Elser  wrote:
>
>
>
> On 6/25/18 11:34 PM, Andy LoPresto wrote:
>
> Hi folks,
>
> I do not want to start a long-running argument or entrenched battle.
> However, having just performed the RM duties for the latest release, I
> believe I have identified a resource inefficiency in the fact that we
> generate, upload, host, and distribute two compressed archives of the
> binary which are functionally equivalent. For 1.7.0, both the .tar.gz
> and .zip files are 1.2 GB (1_224_352_000 bytes for tar.gz vs.
> 1_224_392_000 bytes for zip). The time to build and sign these is
> substantial, but the true cost comes in uploading and hosting them.
> While the fabled extension registry will save all of us from this
> burden, it isn’t arriving tomorrow, and I think we could drastically
> improve this before the next release.
>
> I have no personal preference between the two formats. In earlier days,
> there were platform inconsistencies and the tools weren’t available on
> all systems, but now they are pretty standard for all users. This [1]
>
> is
>
> an interesting article I found which had some good info on the origins,
> and here are some additional resources for anyone interested [2][3]. I
> don’t care which we pick, but I propose removing one of the options for
> the build going forward (toolkit as well).
>
> That said, if someone has a good reason that both are necessary, I
>
> would
>
> love to hear it. I didn’t find anything on the Apache Release Policy
> which stated we must offer both, but maybe I missed it. Thanks.
>
>
> I'm not aware of any ASF policy. I think it mostly stems from default
> convention you get out of the maven-assembly-plugin.
>
> [1] https://itsfoss.com/tar-vs-zip-vs-gz/
> [2] https://superuser.com/a/1257441/40003
> [3] https://superuser.com/a/173995/40003
> [4] https://www.apache.org/legal/release-policy.html#artifacts
>
>
> Andy LoPresto
> alopre...@apache.org 
> /alopresto.apa...@gmail.com  >/
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
>
>
>
>
>
>