Re: Dockerfile and Docker Hub Management
Thanks, Aldrin. I'm a +1 for a separate NiFi Docker project and release process, allowing improvements to the containerization of NiFi artifacts to be made independently from the artifacts (and their source files). I have observed many community members maintaining their own NiFi Dockerfiles and Docker Compose files, and it would be nice if the most reusable variant(s) of these could be well maintained in one location on an independent schedule. I also agree a separate repository would be cleaner and preferable that a subtree of the existing NiFi repository. Thanks, Kevin On 9/21/17, 11:12, "Aldrin Piri" wrote: Hey folks, ** This message turned out to be more detailed than anticipated. In summary, I propose consolidating Docker/container work with a separate release process outside of the repository they are packaging. Full thoughts and background follow. Any input would be appreciated! --- I've been working through providing some additional Docker capabilities for the project and wanted to share some thoughts as well as possible plans to help us be a bit more nimble and responsive to curating Dockerfiles and their respective images on DockerHub. As a bit of context, we currently have the core NiFi project captured in two Dockerfiles, one that is used in conjunction with a Maven plugin for creating an image during the NiFi build (dockermaven), and another that is used for building tagged releases on Docker Hub (dockerhub). Both of these artifacts, currently, reside in a nifi-docker project and are activated via Maven profile, (-P docker). We've seen at times that this is a very coupled process and limits our flexibility. For instance, we had an ill-placed 'chown' which caused a duplicating layer and causes our image to be doubly large. While this has been remedied, given current release processes, this is included with the core nifi release and we have been unable to rectify that issue. Another issue is a very specific sequence of actions that needs to happen with the current release for artifacts to be triggered correctly in Docker Hub. This can be seen in Section 6 of the release guide [1]. While there are ways to rectify this if the timing isn't quite right and/or an error is made, it can impose an additional burden on the INFRA team to facilitate these requests as there currently is no capability for PMCs to manage their Docker repositories directly. Ultimately, I think we should consider a separate release process for NiFi Docker, and any associated efforts that may relate to those files. In this context, NiFi is encompassing of all projects/efforts in the project. Additional efforts could comprise of examples of configuring NiFi to be secured or clustered, receive data from MiNiFi instances, or using Docker Compose or other orchestration frameworks. I have also noticed a number of different areas across our work that are using Docker for integration testing purposes. With some planning and coordination, we could likely consolidate many of these core resources/templates to allow us to reuse them across efforts. I believe there are two approaches from an organizational standpoint that let us execute on the separate release process effectively: 1.) Condense all Docker artifacts into the current NiFi repository [2]. We update our release for NiFi to exclude the Docker subtree to carry out our normal release flow and provide the build/tooling for the Docker subtree to be released on its own. 2.) Establish a new git repository to handle Docker and any other containerization efforts and migrate all existing resources into a file structure that makes sense. My inclination is toward (2). Regardless of path chosen above, this frees us to handle updates and improvements to container efforts when needed. Any time we wanted to release updates to Docker images, we could perform a separate release on either the subtree of (1) or the repository of (2) and reference the associated latest artifacts of NiFi. If you've made it this far, thanks for working through the wall of text and would appreciate any thoughts or comments. [1] http://nifi.apache.org/release-guide.html [2] https://git-wip-us.apache.org/repos/asf?p=nifi.git
Re: Dockerfile and Docker Hub Management
I'd love to retire https://github.com/mkobit/docker-nifi as well since you have started providing your own images. On Thu, Sep 21, 2017 at 11:42 AM Kevin Doran wrote: > Thanks, Aldrin. > > I'm a +1 for a separate NiFi Docker project and release process, allowing > improvements to the containerization of NiFi artifacts to be made > independently from the artifacts (and their source files). I have observed > many community members maintaining their own NiFi Dockerfiles and Docker > Compose files, and it would be nice if the most reusable variant(s) of > these could be well maintained in one location on an independent schedule. > > I also agree a separate repository would be cleaner and preferable that a > subtree of the existing NiFi repository. > > Thanks, > Kevin > > On 9/21/17, 11:12, "Aldrin Piri" wrote: > > Hey folks, > > ** This message turned out to be more detailed than anticipated. In > summary, I propose consolidating Docker/container work with a separate > release process outside of the repository they are packaging. Full > thoughts and background follow. Any input would be appreciated! > > --- > > I've been working through providing some additional Docker > capabilities for > the project and wanted to share some thoughts as well as possible > plans to > help us be a bit more nimble and responsive to curating Dockerfiles and > their respective images on DockerHub. > > As a bit of context, we currently have the core NiFi project captured > in > two Dockerfiles, one that is used in conjunction with a Maven plugin > for > creating an image during the NiFi build (dockermaven), and another > that is > used for building tagged releases on Docker Hub (dockerhub). Both of > these > artifacts, currently, reside in a nifi-docker project and are > activated via > Maven profile, (-P docker). > > We've seen at times that this is a very coupled process and limits our > flexibility. For instance, we had an ill-placed 'chown' which caused a > duplicating layer and causes our image to be doubly large. While this > has > been remedied, given current release processes, this is included with > the > core nifi release and we have been unable to rectify that issue. > > Another issue is a very specific sequence of actions that needs to > happen > with the current release for artifacts to be triggered correctly in > Docker > Hub. This can be seen in Section 6 of the release guide [1]. While > there > are ways to rectify this if the timing isn't quite right and/or an > error is > made, it can impose an additional burden on the INFRA team to > facilitate > these requests as there currently is no capability for PMCs to manage > their > Docker repositories directly. > > Ultimately, I think we should consider a separate release process for > NiFi > Docker, and any associated efforts that may relate to those files. In > this > context, NiFi is encompassing of all projects/efforts in the project. > Additional efforts could comprise of examples of configuring NiFi to be > secured or clustered, receive data from MiNiFi instances, or using > Docker > Compose or other orchestration frameworks. I have also noticed a > number of > different areas across our work that are using Docker for integration > testing purposes. With some planning and coordination, we could likely > consolidate many of these core resources/templates to allow us to reuse > them across efforts. > > I believe there are two approaches from an organizational standpoint > that > let us execute on the separate release process effectively: > > 1.) Condense all Docker artifacts into the current NiFi repository > [2]. We > update our release for NiFi to exclude the Docker subtree to carry out > our > normal release flow and provide the build/tooling for the Docker > subtree to > be released on its own. > > 2.) Establish a new git repository to handle Docker and any other > containerization efforts and migrate all existing resources into a file > structure that makes sense. > > My inclination is toward (2). > > Regardless of path chosen above, this frees us to handle updates and > improvements to container efforts when needed. Any time we wanted to > release updates to Docker images, we could perform a separate release > on > either the subtree of (1) or the repository of (2) and reference the > associated latest artifacts of NiFi. > > If you've made it this far, thanks for working through the wall of > text and > would appreciate any thoughts or comments. > > [1] http://nifi.apache.org/release-guide.html > [2] https://git-wip-us.apache.org/repos/asf?p=nifi.git > > > >
Re: Dockerfile and Docker Hub Management
Aldrin, +1 to separate repository (bullet #2). The basic premise that Docker releases should happen separate from the main distribution is spot on. I think a separate repository would help keep this separation. I tend to believe that the future of NiFi distributions will be via containerization. Making the Docker components somewhat a standalone initiative will help drive changes and innovation in this area. I'd like to help see Docker become a first-class citizen for distributing, running and upgrading NiFi. Thanks, Adam On Thu, Sep 21, 2017 at 9:11 AM, Aldrin Piri wrote: > Hey folks, > > ** This message turned out to be more detailed than anticipated. In > summary, I propose consolidating Docker/container work with a separate > release process outside of the repository they are packaging. Full > thoughts and background follow. Any input would be appreciated! > > --- > > I've been working through providing some additional Docker capabilities for > the project and wanted to share some thoughts as well as possible plans to > help us be a bit more nimble and responsive to curating Dockerfiles and > their respective images on DockerHub. > > As a bit of context, we currently have the core NiFi project captured in > two Dockerfiles, one that is used in conjunction with a Maven plugin for > creating an image during the NiFi build (dockermaven), and another that is > used for building tagged releases on Docker Hub (dockerhub). Both of these > artifacts, currently, reside in a nifi-docker project and are activated via > Maven profile, (-P docker). > > We've seen at times that this is a very coupled process and limits our > flexibility. For instance, we had an ill-placed 'chown' which caused a > duplicating layer and causes our image to be doubly large. While this has > been remedied, given current release processes, this is included with the > core nifi release and we have been unable to rectify that issue. > > Another issue is a very specific sequence of actions that needs to happen > with the current release for artifacts to be triggered correctly in Docker > Hub. This can be seen in Section 6 of the release guide [1]. While there > are ways to rectify this if the timing isn't quite right and/or an error is > made, it can impose an additional burden on the INFRA team to facilitate > these requests as there currently is no capability for PMCs to manage their > Docker repositories directly. > > Ultimately, I think we should consider a separate release process for NiFi > Docker, and any associated efforts that may relate to those files. In this > context, NiFi is encompassing of all projects/efforts in the project. > Additional efforts could comprise of examples of configuring NiFi to be > secured or clustered, receive data from MiNiFi instances, or using Docker > Compose or other orchestration frameworks. I have also noticed a number of > different areas across our work that are using Docker for integration > testing purposes. With some planning and coordination, we could likely > consolidate many of these core resources/templates to allow us to reuse > them across efforts. > > I believe there are two approaches from an organizational standpoint that > let us execute on the separate release process effectively: > > 1.) Condense all Docker artifacts into the current NiFi repository [2]. We > update our release for NiFi to exclude the Docker subtree to carry out our > normal release flow and provide the build/tooling for the Docker subtree to > be released on its own. > > 2.) Establish a new git repository to handle Docker and any other > containerization efforts and migrate all existing resources into a file > structure that makes sense. > > My inclination is toward (2). > > Regardless of path chosen above, this frees us to handle updates and > improvements to container efforts when needed. Any time we wanted to > release updates to Docker images, we could perform a separate release on > either the subtree of (1) or the repository of (2) and reference the > associated latest artifacts of NiFi. > > If you've made it this far, thanks for working through the wall of text and > would appreciate any thoughts or comments. > > [1] http://nifi.apache.org/release-guide.html > [2] https://git-wip-us.apache.org/repos/asf?p=nifi.git >
Re: Dockerfile and Docker Hub Management
It seems like the separate release process is the one of preference. I have created a JIRA issue (https://issues.apache.org/jira/browse/NIFI-4425) to capture some of the sentiment from this thread and to subsequently track the efforts once we start executing upon them. Feel free to add any additional comments/thoughts to that ticket. Thanks! --aldrin On Thu, Sep 21, 2017 at 1:44 PM, Adam Taft wrote: > Aldrin, > > +1 to separate repository (bullet #2). The basic premise that Docker > releases should happen separate from the main distribution is spot on. I > think a separate repository would help keep this separation. > > I tend to believe that the future of NiFi distributions will be via > containerization. Making the Docker components somewhat a standalone > initiative will help drive changes and innovation in this area. I'd like > to help see Docker become a first-class citizen for distributing, running > and upgrading NiFi. > > Thanks, > > Adam > > > > On Thu, Sep 21, 2017 at 9:11 AM, Aldrin Piri wrote: > > > Hey folks, > > > > ** This message turned out to be more detailed than anticipated. In > > summary, I propose consolidating Docker/container work with a separate > > release process outside of the repository they are packaging. Full > > thoughts and background follow. Any input would be appreciated! > > > > --- > > > > I've been working through providing some additional Docker capabilities > for > > the project and wanted to share some thoughts as well as possible plans > to > > help us be a bit more nimble and responsive to curating Dockerfiles and > > their respective images on DockerHub. > > > > As a bit of context, we currently have the core NiFi project captured in > > two Dockerfiles, one that is used in conjunction with a Maven plugin for > > creating an image during the NiFi build (dockermaven), and another that > is > > used for building tagged releases on Docker Hub (dockerhub). Both of > these > > artifacts, currently, reside in a nifi-docker project and are activated > via > > Maven profile, (-P docker). > > > > We've seen at times that this is a very coupled process and limits our > > flexibility. For instance, we had an ill-placed 'chown' which caused a > > duplicating layer and causes our image to be doubly large. While this > has > > been remedied, given current release processes, this is included with the > > core nifi release and we have been unable to rectify that issue. > > > > Another issue is a very specific sequence of actions that needs to happen > > with the current release for artifacts to be triggered correctly in > Docker > > Hub. This can be seen in Section 6 of the release guide [1]. While > there > > are ways to rectify this if the timing isn't quite right and/or an error > is > > made, it can impose an additional burden on the INFRA team to facilitate > > these requests as there currently is no capability for PMCs to manage > their > > Docker repositories directly. > > > > Ultimately, I think we should consider a separate release process for > NiFi > > Docker, and any associated efforts that may relate to those files. In > this > > context, NiFi is encompassing of all projects/efforts in the project. > > Additional efforts could comprise of examples of configuring NiFi to be > > secured or clustered, receive data from MiNiFi instances, or using Docker > > Compose or other orchestration frameworks. I have also noticed a number > of > > different areas across our work that are using Docker for integration > > testing purposes. With some planning and coordination, we could likely > > consolidate many of these core resources/templates to allow us to reuse > > them across efforts. > > > > I believe there are two approaches from an organizational standpoint that > > let us execute on the separate release process effectively: > > > > 1.) Condense all Docker artifacts into the current NiFi repository [2]. > We > > update our release for NiFi to exclude the Docker subtree to carry out > our > > normal release flow and provide the build/tooling for the Docker subtree > to > > be released on its own. > > > > 2.) Establish a new git repository to handle Docker and any other > > containerization efforts and migrate all existing resources into a file > > structure that makes sense. > > > > My inclination is toward (2). > > > > Regardless of path chosen above, this frees us to handle updates and > > improvements to container efforts when needed. Any time we wanted to > > release updates to Docker images, we could perform a separate release on > > either the subtree of (1) or the repository of (2) and reference the > > associated latest artifacts of NiFi. > > > > If you've made it this far, thanks for working through the wall of text > and > > would appreciate any thoughts or comments. > > > > [1] http://nifi.apache.org/release-guide.html > > [2] https://git-wip-us.apache.org/repos/asf?p=nifi.git > > >