Hi Chris,
Thanks for the summary of your findings.
I agree that we should clear up our process for generating Docker images and
that the process should be consistent as possible across NiFi, MiNiFi Java,
Registry, and Toolkit given they are all part of the same repository and maven
project. This will clear up confusion and improve maintainability.
I think both of these methods are important and useful:
1. building from downloaded convenience binaries for previously released
versions
2. building from a local assembly output
That said, I think we could probably consolidate to a single Dockerfile that
takes the binary location as an argument that supports either a URL or local
file path (or a version which could default to the current behavior of
inferring a URL location). This would let us continue to support the
flexibility we have today while maintaining a single Docker file rather than
dockermaven and dockerhub variants. If you and others on the list agree, I can
open up a Jira summarizing what needs to be done.
Thanks,
Kevin
> On Nov 5, 2021, at 05:00, Chris Sampson <chris.samp...@naimuri.com.INVALID>
> wrote:
>
> So the good news is that I realised what I was doing wrong yesterday - I'd
> started a local installation after building the RC, then not shut that down
> before booting up the Docker instance, so the username/password I was
> trying to use from the Docker logs was wrong against the local
> installation, d'oh!
>
> Having corrected that this morning, I've successfully booted up and logged
> into a Docker instance built using the RC (with my NIFI-8779 changes so I
> could build from the Dev server instead of the main Apache Archive).
>
> The reason the dockermaven build works is that it uses the locally built
> archive files (i.e. you have to do a full Maven build locally first to
> generate the ZIP files and then create teh Docker image). The
> dockerhub build actually downloads published artifacts from the Apache
> servers - to do this the Dockerfile needs to know the exact path to the
> artifacts it's trying to download, of course.
>
> There was a question recently in one of the Slack channels about whether
> both of these builds (dockerhub and dockermaven) were still valid/needed -
> I think Joe replied that he wasn't sure (Docker not being an area he
> ventures into much, which is fair enough). It may be worth considering
> whether these two modules are both still needed or can be rationalised and
> if the latter, which approach should be used - download from the archive
> server or build from local (both have dis/advantages, but the more
> "official" way is arguably to download from the server).
>
> Also maybe worth noting:
> * NiFi Registry only has the "build from local" Dockerfile setup
> * MiniFi (Java) has both local and archive download options, but the latter
> cannot be used to build an image from the Dev server (the BASE_URL cannot
> be changed via a build arg/env var... this is the issue addressed by
> NIFI-8779 for NiFi)
> * NiFi Toolkit has both local and archive download options, but are located
> in a single assembly module (instead of split into two like NiFi and MiniFi)
>
> Assuming the "nifi/<version>" path will be used for the actual release
> artifacts, this shouldn't be a blocker, but it's one to note/remember when
> the time comes (assuming the "dockerhub" module is what's used to publish
> the "apache/nifi" image to Docker Hub that is).
>
>
> ---
> *Chris Sampson*
> IT Consultant
> chris.samp...@naimuri.com
>
>
> On Fri, 5 Nov 2021 at 03:34, David Handermann <exceptionfact...@apache.org>
> wrote:
>
>> Chris,
>>
>> I am running through several verification steps, but I built 1.15.0 RC 3
>> from sources and then ran "mvn install -pl :dockermaven -P docker" to build
>> the Docker image. When starting with the following command, I was able to
>> use the generated username and password to access the running NiFi
>> container:
>>
>> docker run --name nifi -p 8443:8443 apache/nifi:1.15.0-dockermaven
>>
>> If you are still having issues, it would be helpful to provide any logs or
>> error messages you are seeing.
>>
>> Regards,
>> David Handermann
>>
>> On Thu, Nov 4, 2021 at 8:10 PM Kevin Doran <kdoran.apa...@gmail.com>
>> wrote:
>>
>>> Oh meant to send a link to this too:
>>>
>>>> ARG
>>>
>> NIFI_BINARY_PATH=${NIFI_BINARY_PATH:-/nifi/${NIFI_VERSION}/nifi-${NIFI_VERSION}-bin.zip}
>>>
>>>
>>>
>> https://github.com/apache/nifi/blob/373498445fe589e2d4855a0730fbb9127f0b4452/nifi-docker/dockerhub/Dockerfile#L30
>>>
>>>> On Nov 4, 2021, at 9:09 PM, Kevin Doran <kdoran.apa...@gmail.com>
>> wrote:
>>>>
>>>> Hi Joe and Chris,
>>>>
>>>> Our Dockerfile that we use to build the Dockerhub image defaults to
>>> looking for 1.15.0 instead of NiFi-1.15.0, but it is a variable that we
>> can
>>> override, so this is easy to workaround incase the release folder does
>>> change. Agree its nice to keep the tree structure consistent when we can.
>>>>
>>>> I’m about to do my verification and will also check the single user
>> with
>>> the docker image as part of that.
>>>>
>>>> Cheers,
>>>> Kevin
>>>>
>>>>> On Nov 4, 2021, at 6:44 PM, Joe Witt <joe.w...@gmail.com> wrote:
>>>>>
>>>>> Chris,
>>>>>
>>>>> Yeah I should have just put it in 1.15.0 instead of nifi/1.15.0.
>>>>> Should generally not really matter so the docker angle there is
>>>>> interesting. Not sure why our docker images would have any
>>>>> relationship to our dist/dev storage. But when I move them into the
>>>>> release folder I can try to ensure I place them only in 1.15.0 instead
>>>>> of nifi/1.15.0.
>>>>>
>>>>> That directory the prov stuff makes does linger and can be annoying so
>>>>> thanks for tackling that. Saw the PR. Will take a look soon if
>>>>> nobody else has.
>>>>>
>>>>> Will keep an eye on your findings related to docker builds not working
>>>>> with username/password things. Hopefully others can chime in there.
>>>>>
>>>>> Thanks
>>>>> Send
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Nov 4, 2021 at 2:06 PM Chris Sampson
>>>>> <chris.samp...@naimuri.com.invalid> wrote:
>>>>>>
>>>>>> Worryingly, when I do get the Docker Image to build (further changes
>>> to the
>>>>>> Dockerfile), the auto-generated username and password from the
>> startup
>>> logs
>>>>>> aren't being accepted for login via my browser.
>>>>>>
>>>>>> I'll try to spend a little more time looking at this (but await input
>>> on my
>>>>>> earlier question/concern also).
>>>>>>
>>>>>>
>>>>>> ---
>>>>>> *Chris Sampson*
>>>>>> IT Consultant
>>>>>> chris.samp...@naimuri.com
>>>>>>
>>>>>>
>>>>>> On Thu, 4 Nov 2021 at 20:47, Chris Sampson <
>> chris.samp...@naimuri.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I've got most of the way through the release check process in order
>> to
>>>>>>> vote for 1.15.0, but I wanted to check on a change in the
>> distribution
>>>>>>> release artifacts.
>>>>>>>
>>>>>>> For 1.14.0, the Dev artifacts were located at:
>>>>>>> https://dist.apache.org/repos/dist/dev/nifi/1.14.0/*
>>>>>>> For 1.15.0, the Dev artifacts are located at:
>>>>>>> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.15.0/*
>>>>>>>
>>>>>>> i.e. there's been a change of directory/path from <version> to
>>>>>>> nifi-<version>.
>>>>>>>
>>>>>>> The reason I raise this is that I can no longer build a Docker Image
>>> using
>>>>>>> the dockerhub/DockerBuild.sh script because it cannot find the
>>> artifacts to
>>>>>>> download. This may not be a problem if the path change is only for
>>> the Dev
>>>>>>> artifacts, but if the same change is going to happen for the
>> released
>>>>>>> artifacts, then the apache/nifi image (and presumably the
>>>>>>> apache/nifi-registry, apache/nifi-toolkit and any minifi)
>> convenience
>>>>>>> images will no longer be possible as part of the Release, which will
>>> likely
>>>>>>> be an issue for a number of users that have deployments using these
>>> images.
>>>>>>>
>>>>>>> I spotted this after rebasing my outstanding PR [1] for NIFI-8779
>> [2]
>>>>>>>
>>>>>>>
>>>>>>> Additionally, I noted NIFI-9366 [3] after an unwanted directory was
>>>>>>> created by the unit tests executed during building and testing
>>> 1.15.0-RC3.
>>>>>>> I raised a PR [4] - this is a minor issue and not a reason to block
>>> any
>>>>>>> release.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> [1] https://github.com/apache/nifi/pull/5213
>>>>>>> [2] https://issues.apache.org/jira/browse/NIFI-8779
>>>>>>>
>>>>>>> [3] https://issues.apache.org/jira/browse/NIFI-9366
>>>>>>> [4] https://github.com/apache/nifi/pull/5510
>>>>>>>
>>>>>>> ---
>>>>>>> *Chris Sampson*
>>>>>>> IT Consultant
>>>>>>> chris.samp...@naimuri.com
>>>>>>>
>>>>>>>
>>>>>>> On Sat, 30 Oct 2021 at 01:52, Joe Witt <joew...@apache.org> wrote:
>>>>>>>
>>>>>>>> Otto
>>>>>>>>
>>>>>>>> Got ya. Yeah it was this way in 1.14 as well. And to be clear
>> with
>>> every
>>>>>>>> release what we are actually voting upon is the source release.
>> Now
>>> the
>>>>>>>> source release includes nifi, minifi, nifi registry, stateless nifi
>>> and
>>>>>>>> toolkits among all the other having always been there goodies.
>>>>>>>>
>>>>>>>> Some of these things we make available in the form of convenience
>>> binaries
>>>>>>>> to make it easier on folks to consume.
>>>>>>>>
>>>>>>>> I think you dont need to do any verification you would not have
>> done
>>>>>>>> before.
>>>>>>>>
>>>>>>>> But I do hope folks help maintain various ways of easing more folks
>>>>>>>> knowing
>>>>>>>> what to vet with a given release
>>>>>>>>
>>>>>>>> On Fri, Oct 29, 2021 at 5:34 PM Otto Fowler <
>> ottobackwa...@gmail.com
>>>>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> This release, we are verifying not only Nifi, but also Minifi
>> Java.
>>> At
>>>>>>>>> least that is my understanding.
>>>>>>>>>
>>>>>>>>> This would be my first time having *anything* to do with minifi,
>>> i’ve
>>>>>>>> not
>>>>>>>>> even run it before.
>>>>>>>>>
>>>>>>>>> As such, I think the RC validation guide needs to be update to
>>> include
>>>>>>>> the
>>>>>>>>> information about now validating nifi and minifi together.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> From: Joe Witt <joew...@apache.org> <joew...@apache.org>
>>>>>>>>> Reply: dev@nifi.apache.org <dev@nifi.apache.org> <
>>> dev@nifi.apache.org>
>>>>>>>>> Date: October 25, 2021 at 11:59:39
>>>>>>>>> To: dev@nifi.apache.org <dev@nifi.apache.org> <
>> dev@nifi.apache.org>
>>>>>>>>> Subject: [discuss] NiFi 1.15
>>>>>>>>>
>>>>>>>>> Team,
>>>>>>>>>
>>>>>>>>> I thought I had already started such a thread but I dont see it so
>>> here
>>>>>>>>> goes...
>>>>>>>>>
>>>>>>>>> We have made a ton of progress again and there are some super
>>>>>>>>> important fixes as well. It is definitely time to kick out a 1.15.
>>>>>>>>> My intent will be to attempt to pull together an RC this week.
>>>>>>>>> Haven't done the analysis yet of what is hanging out there but
>> will
>>> do
>>>>>>>>> so. A quick look at all the features and fixes already landed
>> though
>>>>>>>>> make it clear we have more than enough to work with.
>>>>>>>>>
>>>>>>>>> Lets please use this thread for coordination on the RC rather than
>>> it
>>>>>>>>> becoming the wish list. We have new features/fixes arriving all
>> the
>>>>>>>>> time - those can be addressed in normal channels.
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>
>>>
>>>
>>
