[RESULT][VOTE] Release Apache NiFi Flow Design System 0.3.0

[Scott Aslan]

Apache NiFi Community,

I am pleased to announce that including my +1 (binding) vote the 0.3.0
release of Apache NiFi Flow Design System passes with
4 +1 (binding) votes
5 +1 (non-binding) votes
0 0 votes
0 -1 votes

Thanks to all who helped make this release possible.

Here is the PMC vote thread:
https://lists.apache.org/thread/r8vd4mln6nmcb3pxw6kpwkc3zco19mw6

Re: Doco, PutSolrContentStream

[Andrew Lim]

Hi Dwane,

Thanks for finding and reporting those documentation errors. I filed a Jira [1] 
to fix those.

It looks like the change to the default value of 
nifi.provenance.repository.rollover.time was made in 1.12.0 [2]. I will see if 
we can improve the docs to give more context to why this was done as part of 
[1].

-Drew


[1] https://issues.apache.org/jira/browse/NIFI-9642 

[2] https://issues.apache.org/jira/browse/NIFI-7339 



> On Jan 28, 2022, at 7:56 AM, Nathan Gough  wrote:
> 
> Hi Dwane,
> 
> I've created a Jira issue to test and rectify the Solr + ZooKeeper issue: 
> https://issues.apache.org/jira/browse/NIFI-9641 
> 
> 
> Thanks for the report!
> Nathan
> 
> On Fri, Jan 28, 2022 at 7:22 AM Dwane Hall  > wrote:
> Hey NiFi community I hope all is well with everyone wherever they may be.  I 
> recently updated our NiFi instances from 1.11.4 to 1.15.3 and have made a few 
> observations from this process worth mentioning.
>  
> Some minor documentation inconsistencies
> A couple of the default values appear to have changed in nifi.properties 
> through versions (listed below are the old and new values along with links to 
> the documentation). 
>  
> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#write-ahead-flowfile-repository
>  
> 
> “The FlowFile Repository checkpoint interval. The default value is 2 mins.” 
> [new default value is 20 secs]
> 1.11.4 nifi.flowfile.repository.checkpoint.interval=2 mins
> 11.15.3 nifi.flowfile.repository.checkpoint.interval=20 secs
>  
> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#persistent-provenance-repository-properties
>  
> 
> “The amount of time to wait before rolling over the latest data provenance 
> information so that it is available in the User Interface. The default value 
> is 30 secs.”
> https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#system-properties 
> 
> “If processing a high volume of events, change 
> nifi.provenance.repository.rollover.time from a default of 30 secs to 1 min 
> and ...” [The new default value is 10 min].
> 1.11.4 nifi.provenance.repository.rollover.time=30 sec
> 1.15.3 nifi.provenance.repository.rollover.time=10 min
> This seems to be a significant change was there any reason for this new 
> default setting I was unable to find documentation referencing the increase?
>  
> PutSolrContentStream processor issues
>  
> Secondly after a successful upgrade I noticed our use of the 
> PutSolrContentStream processor had broken.  Looking through the processor 
> code there was an upgrade to the SolrJ client and a commit in March 2020 (and 
> referenced below) that appears to prevent nested zk chroot paths for 
> SolrCloud connections (i.e. the zookeeper connection string is truncated).
>  
> SolrUtils.java (nifi/SolrUtils.java at master · apache/nifi · GitHub 
> )
> The commit of intrest regarding the new process for initiating a 
> CloudSolrClient in SolrJ 
> https://github.com/apache/nifi/commit/9b4292024be6fae188cb1efa3a07dc9489e9a5b4#diff-13320e5b198f236cea296fb01cb7376755d65c444678e781fa0940c2a28db88b
>  
> 
>  
> For a nested Solr path "/solr/PROD", "/solr/DEV", "/solr/DR" … the string is 
> truncated to the base path only i.e. “/solr” (this is only an issue for 
> nested chroots)
>  
> The code of interest is here in the SolrUtils.java class
>  
> if 
> (SOLR_TYPE_STANDARD.getValue().equals(context.getProperty(SOLR_TYPE).getValue()))
>  {
> return new HttpSolrClient(solrLocation, httpClient);
> return new 
> HttpSolrClient.Builder(solrLocation).withHttpClient(httpClient).build();
> } else {
> // CloudSolrClient.Builder now requires a List of ZK addresses 
> and znode for solr as separate parameters
> final String zk[] = solrLocation.split("/");
> final List zkList = Arrays.asList(zk[0].split(","));
> String zkRoot = "/";
> if (zk.length > 1 && ! zk[1].isEmpty()) {
> zkRoot += zk[1];
> }
>   
> 
>  
>
> I think the issue can be resolved by changing this line of code which should 
> captur

Re: apache/nifi convenience Docker Image - publish JDK 11 version?

[Kevin Doran]

Hi Chris,

I forgot to mention - Bryan Bende was able to help me with this, so you should 
have edit permissions in Confluence now.

Thanks,
Kevin

> On Jan 13, 2022, at 12:09, Chris Sampson  
> wrote:
> 
> Thanks Kevin, that's great.
> 
> My username is "chriss".
> 
> 
> ---
> *Chris Sampson*
> IT Consultant
> chris.samp...@naimuri.com
> 
> 
> On Thu, 13 Jan 2022 at 16:38, Kevin Doran  wrote:
> 
>> Thanks, Chris! I’ve incorporated these ideas into the Confluence page.
>> I’ll look into getting you edit permissions - what is your username there?
>> 
>>> On Dec 18, 2021, at 13:35, Chris Sampson 
>> wrote:
>>> 
>>> Kevin,
>>> 
>>> Good idea to write up a list of ideas - unfortunately I don't appear to
>>> have permission to edit the page, but a few other ideas are:
>>> 
>>>  - Provide an "alpine" based image (could be the
>>>  "openjdk:-alpine" image used as a base) to reduce image size
>> and
>>>  O/S footprint of the container
>>>  - Change the Docker start scripts to allow for all nifi.properties (and
>>>  other config file properties) to be set by Environment Variables - this
>>>  could be a case of using a common Env Var prefix to and iterating
>> through
>>>  them within the script, e.g. anything that is NIFI_* is assumed to be a
>>>  nifi.properties entry, etc. - this would make the convenience image
>> all the
>>>  more convenient for people and follow suit with some other applications
>>>  using Docker wrappers
>>>  - Docker-focussed default config, for example logback.xml that directs
>>>  output to STDOUT/STDERR by default, possibly in JSON format, allowing
>>>  easier out-of-the-box use of the logs in Docker-based environments that
>>>  already process Docker log output
>>>  - Allow for the nifi.properties/config file changes during start.sh to
>>>  be skipped (some people inject the config files into their containers
>> but
>>>  want them to be read-only, which then means the start.sh script fails
>>>  because the files can't be edited)
>>>  - Change the default config file locations within the Docker images,
>>>  e.g. the flow.xml.gz and other system generated config that should be
>>>  persisted through container restart would be better in a different
>> folder
>>>  to things like nifi.properties, then the directory can be externalised
>> as a
>>>  Volume (without running into read-only/mutability issues like the
>> start.sh
>>>  point above)
>>>  - Make it easier for Docker-based instances to join an existing cluster
>>>  - this might be a case of relocating some of the files to make better
>> use
>>>  of ephemeral vs. persistent directories/Volumes and/or having a
>>>  configurable mechanism that allows for the cluster config files to be
>>>  deleted during startup if the instance expects to join a cluster but
>> cannot
>>>  because it is unable to overwrite existing files with those being
>> offered
>>>  by the cluster (not sure how much of an issue this still is after some
>>>  recent changes)
>>> 
>>> Please feel free to add any/all of the above to the Confluence page that
>>> you think would be worthwhile to add to the pile for consideration.
>>> 
>>> 
>>> ---
>>> *Chris Sampson*
>>> IT Consultant
>>> chris.samp...@naimuri.com
>>> 
>>> 
>>> On Fri, 17 Dec 2021 at 18:01, Kevin Doran  wrote:
>>> 
 Hi Chris,
 
 I like this idea and have been thinking of a number of other
>> improvements
 we could make for container images of Apache NiFi, including (I think
 discussed on another thread) consolidating Dockerfiles for dockerhub /
 dockermaven, as the only significant difference is where the nifi
>> assembly
 archive comes from, arm64 images that run native on ARM platforms,
 including newer M1 Macs (requires some minor changes to NiFi source
>> itself
 to compile).
 
 I’ve started a Wiki page [1] to collect ideas, and once we reach
>> consensus
 on the improvements we’d like to make and possible approaches, we can
>> turn
 these into Jiras.
 
 [1]
 
>> https://cwiki.apache.org/confluence/display/NIFI/NiFi+Docker+Container+Improvements
 
 Thanks!
 Kevin
 
> On Nov 26, 2021, at 07:46, Chris Sampson > .INVALID>
 wrote:
> 
> I'm not sure whether I've seen this discussed previously, but I thought
 it
> worth asking whether we could/should publish a JDK 11 based version of
 the
> apache/nifi Docker Image to Docker Hub as part of the release process
>> in
> future?
> 
> I realise NiFi 1.x is/will remain based on JDK 8 but with all the work
 gone
> into enabling JDK 11 compilation and deployment, along with the planned
> move to JDK 11 in the future for NiFi 2.x, it would seem sensible to
 start
> making it easier for people to move towards using JDK 11 now that it's
> available.
> 
> The Dockerfiles used to build NiFi (I've not looked at Toolkit, MiNiFi
>> or
> Registry) can be given a build ARG to use different base
>> images/versions,
>