Re: NiFi Registry metadata DB lost data, can we restore from disk?
and apologies for the full email signature...
Dave
On Tue, Jul 30, 2024 at 12:49 PM David Early
wrote:
> I am going to answer my own question.
>
> Yes, you can for a default H2 database, at least we have not had a
> catastrophic error since doing the following.
>
> -
> In our case, all the flows were still in flow_storage AND nothing was lost
> and the buckets were still defined in the H2 database.
>
> Before you begin, you will need to go into the flow_storage and figure out
> what you have:
>
> # find ./8f0d9b79-b329-43d6-ad65-2a0db627492e -type f
>
> ./8f0d9b79-b329-43d6-ad65-2a0db627492e/622e1509-3a65-420d-95be-4b8214e65f3c/6/6.snapshot
>
> ./8f0d9b79-b329-43d6-ad65-2a0db627492e/622e1509-3a65-420d-95be-4b8214e65f3c/3/3.snapshot
>
> ./8f0d9b79-b329-43d6-ad65-2a0db627492e/622e1509-3a65-420d-95be-4b8214e65f3c/9/9.snapshot
>
> ./8f0d9b79-b329-43d6-ad65-2a0db627492e/622e1509-3a65-420d-95be-4b8214e65f3c/10/10.snapshot
>
> ./8f0d9b79-b329-43d6-ad65-2a0db627492e/622e1509-3a65-420d-95be-4b8214e65f3c/5/5.snapshot
>
> ./8f0d9b79-b329-43d6-ad65-2a0db627492e/622e1509-3a65-420d-95be-4b8214e65f3c/4/4.snapshot
>
> ./8f0d9b79-b329-43d6-ad65-2a0db627492e/622e1509-3a65-420d-95be-4b8214e65f3c/8/8.snapshot
>
> ./8f0d9b79-b329-43d6-ad65-2a0db627492e/622e1509-3a65-420d-95be-4b8214e65f3c/2/2.snapshot
>
> ./8f0d9b79-b329-43d6-ad65-2a0db627492e/622e1509-3a65-420d-95be-4b8214e65f3c/7/7.snapshot
>
> ./8f0d9b79-b329-43d6-ad65-2a0db627492e/622e1509-3a65-420d-95be-4b8214e65f3c/1/1.snapshot
>
> ./8f0d9b79-b329-43d6-ad65-2a0db627492e/622e1509-3a65-420d-95be-4b8214e65f3c/11/11.snapshot
>
> ./8f0d9b79-b329-43d6-ad65-2a0db627492e/9e2fce9a-26c4-4b1c-8139-8ae0f828313f/3/3.snapshot
>
> ./8f0d9b79-b329-43d6-ad65-2a0db627492e/9e2fce9a-26c4-4b1c-8139-8ae0f828313f/5/5.snapshot
>
> ./8f0d9b79-b329-43d6-ad65-2a0db627492e/9e2fce9a-26c4-4b1c-8139-8ae0f828313f/4/4.snapshot
>
> ./8f0d9b79-b329-43d6-ad65-2a0db627492e/9e2fce9a-26c4-4b1c-8139-8ae0f828313f/2/2.snapshot
>
> ./8f0d9b79-b329-43d6-ad65-2a0db627492e/9e2fce9a-26c4-4b1c-8139-8ae0f828313f/1/1.snapshot
>
> ./8f0d9b79-b329-43d6-ad65-2a0db627492e/afa05554-1d56-4ec8-b09c-cd1d0001dc70/3/3.snapshot
>
> ./8f0d9b79-b329-43d6-ad65-2a0db627492e/afa05554-1d56-4ec8-b09c-cd1d0001dc70/2/2.snapshot
>
> ./8f0d9b79-b329-43d6-ad65-2a0db627492e/afa05554-1d56-4ec8-b09c-cd1d0001dc70/1/1.snapshot
>
> ./8f0d9b79-b329-43d6-ad65-2a0db627492e/5300c691-dbbf-4f7f-9f02-49c1c01f0188/2/2.snapshot
>
> ./8f0d9b79-b329-43d6-ad65-2a0db627492e/5300c691-dbbf-4f7f-9f02-49c1c01f0188/1/1.snapshot
>
> ./8f0d9b79-b329-43d6-ad65-2a0db627492e/88f6d53c-6066-4fd1-b2f9-9f97c9068124/1/1.snapshot
>
> ./8f0d9b79-b329-43d6-ad65-2a0db627492e/2733df97-2973-430b-8be3-5a68b779021c/2/2.snapshot
>
> ./8f0d9b79-b329-43d6-ad65-2a0db627492e/2733df97-2973-430b-8be3-5a68b779021c/1/1.snapshot
>
> ./8f0d9b79-b329-43d6-ad65-2a0db627492e/0c0045dd-1904-423e-83b0-230543bfc556/1/1.snapshot
>
> The first UUID is the bucket, the second is the flow.
>
> To get the name of the flow from the file, use the following python script:
>
> import json
> import sys
> fn = sys.argv[1]
> with open(fn) as fi:
> jdata = json.load(fi)
> print(fn,jdata['content']['flowSnapshot']['flowContents']['name'])
>
> And run:
>
> cd ./flow_storage
>
> for f in `find . -type f`
> do
> python3 /tmp/getname.py ${f}
> done
>
> This will print out all the names
>
> -
>
> Download the h2 jar binary:
>
> wget
> https://search.maven.org/remotecontent?filepath=com/h2database/h2/2.3.230/h2-2.3.230.jar
> mv
> 'remotecontent?filepath=com%2Fh2database%2Fh2%2F2.3.230%2Fh2-2.3.230.jar'
> h2.jar
>
> Access the H2 datastore:
>
> cd /data/nifi-registry # this is the directory that CONTAINS the database
> directory
>
> Get into the H2 shell:
>
> java -cp ~/h2.jar org.h2.tools.Shell
>
> You will be prompted for a URL. Unless you have changed the default
> (nifi.registry.db.url), it should be this:
>
> jdbc:h2:./database/nifi-registry-primary
>
> From the shell you can view the tables:
>
> SELECT * FROM INFORMATION_SCHEMA.TABLES;
>
>
> In our case, the buckets were all present and correct, so we did not have
> to deal with those but the process is likely similar.
>
>
> You will be working with 3 tables:
>
> BUCKET_ITEM
> FLOW
> FLOW_SNAPSHOT
>
>
> (The order is important)
> Run the following for each snapshot to reinsert into the metadata database:
>
> SLQ1:
> insert into bucket_item
> (ID,NAME,DESCRIPTION,CREATED,MODIFIED,ITEM_TYPE,BUCKET_ID) values
> ('0c0045dd-1904-423e-83b0-230543bfc556','',' appropriate comment>','2024-07-01 00:00:00.000','2024-07-01
> 00:00:00.000','FLOW','8f0d9b79-b329-43d6-ad65-2a0db627492e');
>
> In this case, the created and updated times are the same, you can modify
> these as you see fit. At this point, we just needed them back in, loss of
> the exact times was not relevant.
>
> ID - UUID of the flow
> NAME - Name of the flow, we extracted these from the flow snapshots using
> the script above
> DESCRIPTION - Description that will appear in
Re: NiFi Registry metadata DB lost data, can we restore from disk?
I am going to answer my own question.
Yes, you can for a default H2 database, at least we have not had a
catastrophic error since doing the following.
-
In our case, all the flows were still in flow_storage AND nothing was lost
and the buckets were still defined in the H2 database.
Before you begin, you will need to go into the flow_storage and figure out
what you have:
# find ./8f0d9b79-b329-43d6-ad65-2a0db627492e -type f
./8f0d9b79-b329-43d6-ad65-2a0db627492e/622e1509-3a65-420d-95be-4b8214e65f3c/6/6.snapshot
./8f0d9b79-b329-43d6-ad65-2a0db627492e/622e1509-3a65-420d-95be-4b8214e65f3c/3/3.snapshot
./8f0d9b79-b329-43d6-ad65-2a0db627492e/622e1509-3a65-420d-95be-4b8214e65f3c/9/9.snapshot
./8f0d9b79-b329-43d6-ad65-2a0db627492e/622e1509-3a65-420d-95be-4b8214e65f3c/10/10.snapshot
./8f0d9b79-b329-43d6-ad65-2a0db627492e/622e1509-3a65-420d-95be-4b8214e65f3c/5/5.snapshot
./8f0d9b79-b329-43d6-ad65-2a0db627492e/622e1509-3a65-420d-95be-4b8214e65f3c/4/4.snapshot
./8f0d9b79-b329-43d6-ad65-2a0db627492e/622e1509-3a65-420d-95be-4b8214e65f3c/8/8.snapshot
./8f0d9b79-b329-43d6-ad65-2a0db627492e/622e1509-3a65-420d-95be-4b8214e65f3c/2/2.snapshot
./8f0d9b79-b329-43d6-ad65-2a0db627492e/622e1509-3a65-420d-95be-4b8214e65f3c/7/7.snapshot
./8f0d9b79-b329-43d6-ad65-2a0db627492e/622e1509-3a65-420d-95be-4b8214e65f3c/1/1.snapshot
./8f0d9b79-b329-43d6-ad65-2a0db627492e/622e1509-3a65-420d-95be-4b8214e65f3c/11/11.snapshot
./8f0d9b79-b329-43d6-ad65-2a0db627492e/9e2fce9a-26c4-4b1c-8139-8ae0f828313f/3/3.snapshot
./8f0d9b79-b329-43d6-ad65-2a0db627492e/9e2fce9a-26c4-4b1c-8139-8ae0f828313f/5/5.snapshot
./8f0d9b79-b329-43d6-ad65-2a0db627492e/9e2fce9a-26c4-4b1c-8139-8ae0f828313f/4/4.snapshot
./8f0d9b79-b329-43d6-ad65-2a0db627492e/9e2fce9a-26c4-4b1c-8139-8ae0f828313f/2/2.snapshot
./8f0d9b79-b329-43d6-ad65-2a0db627492e/9e2fce9a-26c4-4b1c-8139-8ae0f828313f/1/1.snapshot
./8f0d9b79-b329-43d6-ad65-2a0db627492e/afa05554-1d56-4ec8-b09c-cd1d0001dc70/3/3.snapshot
./8f0d9b79-b329-43d6-ad65-2a0db627492e/afa05554-1d56-4ec8-b09c-cd1d0001dc70/2/2.snapshot
./8f0d9b79-b329-43d6-ad65-2a0db627492e/afa05554-1d56-4ec8-b09c-cd1d0001dc70/1/1.snapshot
./8f0d9b79-b329-43d6-ad65-2a0db627492e/5300c691-dbbf-4f7f-9f02-49c1c01f0188/2/2.snapshot
./8f0d9b79-b329-43d6-ad65-2a0db627492e/5300c691-dbbf-4f7f-9f02-49c1c01f0188/1/1.snapshot
./8f0d9b79-b329-43d6-ad65-2a0db627492e/88f6d53c-6066-4fd1-b2f9-9f97c9068124/1/1.snapshot
./8f0d9b79-b329-43d6-ad65-2a0db627492e/2733df97-2973-430b-8be3-5a68b779021c/2/2.snapshot
./8f0d9b79-b329-43d6-ad65-2a0db627492e/2733df97-2973-430b-8be3-5a68b779021c/1/1.snapshot
./8f0d9b79-b329-43d6-ad65-2a0db627492e/0c0045dd-1904-423e-83b0-230543bfc556/1/1.snapshot
The first UUID is the bucket, the second is the flow.
To get the name of the flow from the file, use the following python script:
import json
import sys
fn = sys.argv[1]
with open(fn) as fi:
jdata = json.load(fi)
print(fn,jdata['content']['flowSnapshot']['flowContents']['name'])
And run:
cd ./flow_storage
for f in `find . -type f`
do
python3 /tmp/getname.py ${f}
done
This will print out all the names
-
Download the h2 jar binary:
wget
https://search.maven.org/remotecontent?filepath=com/h2database/h2/2.3.230/h2-2.3.230.jar
mv
'remotecontent?filepath=com%2Fh2database%2Fh2%2F2.3.230%2Fh2-2.3.230.jar'
h2.jar
Access the H2 datastore:
cd /data/nifi-registry # this is the directory that CONTAINS the database
directory
Get into the H2 shell:
java -cp ~/h2.jar org.h2.tools.Shell
You will be prompted for a URL. Unless you have changed the default
(nifi.registry.db.url), it should be this:
jdbc:h2:./database/nifi-registry-primary
>From the shell you can view the tables:
SELECT * FROM INFORMATION_SCHEMA.TABLES;
In our case, the buckets were all present and correct, so we did not have
to deal with those but the process is likely similar.
You will be working with 3 tables:
BUCKET_ITEM
FLOW
FLOW_SNAPSHOT
(The order is important)
Run the following for each snapshot to reinsert into the metadata database:
SLQ1:
insert into bucket_item
(ID,NAME,DESCRIPTION,CREATED,MODIFIED,ITEM_TYPE,BUCKET_ID) values
('0c0045dd-1904-423e-83b0-230543bfc556','','','2024-07-01 00:00:00.000','2024-07-01
00:00:00.000','FLOW','8f0d9b79-b329-43d6-ad65-2a0db627492e');
In this case, the created and updated times are the same, you can modify
these as you see fit. At this point, we just needed them back in, loss of
the exact times was not relevant.
ID - UUID of the flow
NAME - Name of the flow, we extracted these from the flow snapshots using
the script above
DESCRIPTION - Description that will appear in registry
CREATED - Time of creation
MODIFIED - Time of last mod
ITEM_TYPE - 'FLOW' (we are not aware of any other values)
BUCKET_ID - UUID of the bucket
SQL2:
insert into flow (ID) values ('0c0045dd-1904-423e-83b0-230543bfc556');
SQL3:
insert into flow_snapshot (FLOW_ID,VERSION,CREATED,CREATED_BY,COMMENTS)
values ('0c0045dd-1904-423e-83b0-230543bfc556',1,'2024-07-01
NiFi Registry metadata DB lost data, can we restore from disk?
We have a registry (1.23.2) that WAS working but for some reason has lost all the metadata in the DB. I was able to get into the H2 DB and see that we do not have any flow informationit just isn't there. However the flows are in the flow_storage directory (flow persistence is the directory, not git). We can find no evidence that the process was restarted or otherwise manipulated, just that the data is now gone. This is a bit of a problem because this is used to bridge configs from one system to another, so I have 2 NiFi instances that reference the same config IDs. Is there a way to restore the flows from flow_storage that would preserve the IDs/versions and allow the existing NiFi systems to see the flows? This was ironically noticed today when we went in to configure a regular backup of the registry DB and flows (we are new to the registry). Dave
Re: Nifi Site to Site error message meaning
Thanks, Isha for the first cut. We have a whole bunch of systems that are in an internal network that are all able to talk using site to site, so we are not unfamiliar with the connection, but after a long session of comparisons and changes, I still cannot this to work. I enabled DEBUG in logback.xml across the boardI now have more debug that I can figure out how to use, but the one thing that recurs: *DEBUG [NiFi Web Server-87] o.a.n.w.s.l.RequestAuthenticationFilter Username not found Remote Address [20.XXX.XXX.XXX]* That is the log from the RECEIVER (Nif with the Input port). The IP address in the log message is the SENDER (nifi with remote process group) (Sorry, not sure how the server/client terminology works here) But no where in the logs does it say what the username is, it just says it is not found, so I am not clear on what is wrong with the username. I have fair experience with certificate generation and we generate our own internal certs using locally generated CA and everything has worked perfectly to this point. I am also pretty familiar with the username format and how to retrieve itI currently have 20+ NiFi systems that are all using site to site with no issues. The server certs have been added on both sides to the local truststore, and I am not getting a PKIX error, very specifically getting the Unauthorized which, as noted, is associated with the hostname. If the hostname was accepted, I would get Forbidden if there were policy issuesthe problem is that I cannot get to that point. The machines are both in azure, but I am using the FQDN of the receiver and routing to the internet. The internal azure networks cannot talk to each other. Any help at this point would be welcome, we have done this before, we are at a loss as to why we cannot do it now. NOTE: After writing the above, I did realize one thing: all our internal NiFi systems are communicating via site to site using common CA certs. The CA cert has been loaded in the truststore and they are all happy to talk to each other as long as the CA's all match. In the example I am talking about above, I have just inserted the server cert (PEM file) into the truststore on both boxes. The systems are OK with this because I don't have a PKIX error, but there may be something at issue with the username?? Dave On Thu, Jun 27, 2024 at 1:29 AM Isha Lamboo wrote: > Hi David, > > > > This typically means one of two things: > >1. that the DN of the (client) certificate does not match the user in >the receiving NiFi instance **exactly**. >Inspect the logs (I think it’s nifi-user.log) to find the DN that the >sending NiFi is providing and edit the username in the receiving NiFi >instance. >2. The user is correct but does not have permissions to send data to >the input port. Each input port needs these permissions set separately. >Right-click on it and click “manage access policies”. In the dropdown box >select “receive site-to-site data” and add the user or group that should be >allowed to send data to this port. > > > > Hope this helps you find the issue. > > > > Regards, > > > > Isha > > > > *Van:* David Early via users > *Verzonden:* donderdag 27 juni 2024 01:33 > *Aan:* users@nifi.apache.org > *Onderwerp:* Nifi Site to Site error message meaning > > > > All, > > > > I am trying to get an HTTP site to site set up, and I have done this a > bunch of times, but I am seeing an error that I have not seen before and > the logs are not helping. > > > > I have gotten PKIX errors and Forbidden, but I am getting an Unauthorized > message: > > > > > > What is this telling me? Where is the problem in the permission chain? > > > > -- > > David > -- David Early, Ph.D. david.ea...@grokstream.com 720-470-7460 Cell
Nifi Site to Site error message meaning
All, I am trying to get an HTTP site to site set up, and I have done this a bunch of times, but I am seeing an error that I have not seen before and the logs are not helping. I have gotten PKIX errors and Forbidden, but I am getting an Unauthorized message: [image: image.png] What is this telling me? Where is the problem in the permission chain? -- David
Re: Why are my journal files so large on node 2 of cluster?
Mark, Yes, it was the flowfile repository. Of all your points, the large attributes is most likely our issue. One of our folks was caching the flowfile (which can be large occasionally) in an attribute ahead of a DB lookup (which would overwrite the contents) and then reinstating the content after merging with the DB lookup. The attribute was not removed after the merge. We have added a couple of items to remove the attribute this morning, but the mere presence of it briefly may be enough to cause the spikes. I have since attached a very large disk and I can see the occasionally spikes: [image: image.png] At 22% on a 512G disk, that is over 110G. What isn't clear is why it is not consistently spiking. We have made some changes to the how long the attribute lives and will monitor over the next couple of days, but likely we will need to cache the contents somewhere and retrieve them later unless someone knows of a better solution here. Thanks for the guidance Dave On Fri, Mar 8, 2024 at 7:05 AM Mark Payne wrote: > Dave, > > When you say that the journal files are huge, I presume you mean the > FlowFile repository? > > There are generally 4 things that can cause this: > - OutOfMemoryError causing the FlowFile repo not to properly checkpoint > - Out of Disk Space causing the FlowFile repo not to properly checkpoint > - Out of open file handles causing the FlowFile repo not to properly > checkpoint > - Creating a lot of huge attributes on your FlowFiles. > > The first 3 situations can be identified by looking for errors in the logs. > For the third one, you need to understand whether or not you’re creating > huge FlowFile attributes. Generally, attributes should be very small - > 100-200 characters or less, ideally. It’s possible that you have a flow > that creates huge attributes but the flow is only running on the Primary > Node, and Node 2 is your Primary Node, which would cause this to occur only > on this node. > > Thanks > -Mark > > > > On Mar 7, 2024, at 9:24 PM, David Early via users > wrote: > > > > I have a massive issue: I have a 2 node cluster (using 5 external > zookeepers on other boxes), and for some reason on node 2 I have MASSIVE > journal files. > > > > I am round robbining data between the nodes, but for some reason node 2 > just fills up. This is the second time this has happened this week. > > > > What should I do? nifi.properties are the same on both systems (except > for local host names).. > > > > Any ideas of what might be causing one node to overload? > > > > Dave > > > > > > -- David Early, Ph.D. david.ea...@grokstream.com 720-470-7460 Cell
Why are my journal files so large on node 2 of cluster?
I have a massive issue: I have a 2 node cluster (using 5 external zookeepers on other boxes), and for some reason on node 2 I have MASSIVE journal files. I am round robbining data between the nodes, but for some reason node 2 just fills up. This is the second time this has happened this week. What should I do? nifi.properties are the same on both systems (except for local host names).. Any ideas of what might be causing one node to overload? Dave
Problems using registry configured with git
Hi all, I accidentally submitted this with no subject a while back, so this is my redo. We are setting up a Nifi Registry (1.16.3 currently) that is tied to git. I get MOST of it working, but run into an issue when I actually try and use a registry entry. Steps: 1. Set up registry 2. Configure git 3. Start version control on a process group 4. Check git (it is entered and committed, as expected) 5. Try and drop the registry item onto the canvas and I get this: [image: image.png] In the registry: [image: image.png] When I had this configured to use the file based storage, it worked fine and I could install things from the registry. In addition, there is this log message: 2023-02-09 21:56:05,784 WARN [NiFi Registry Web Server-16] o.a.n.registry.service.RegistryService The specified bucket id [8ef77681-b71d-4687-b627-29e0738d979b] does not exist. The problem is that is an OLD bucket I reset the registry completely and even searched the flow.xml for the 2 systems trying to use the registry and I am STILL getting this error. I even cleaned out the git repo. I am using the same repo, but deleted everything and commited the delete before starting the registry up. I am a bit lostany thoughts/ideas? David
[no subject]
Hi all, We are setting up a Nifi Registry (1.16.3 currently) that is tied to git. I get MOST of it working, but run into an issue when I actually try and use a registry entry. Steps: 1. Set up registry 2. Configure git 3. Start version control on a process group 4. Check git (it is entered and committed, as expected) 5. Try and drop the registry item onto the canvas and I get this: [image: image.png] In the registry: [image: image.png] When I had this configured to use the file based storage, it worked fine and I could install things from the registry. In addition, there is this log message: 2023-02-09 21:56:05,784 WARN [NiFi Registry Web Server-16] o.a.n.registry.service.RegistryService The specified bucket id [8ef77681-b71d-4687-b627-29e0738d979b] does not exist. The problem is that is an OLD bucket I reset the registry completely and even searched the flow.xml for the 2 systems trying to use the registry and I am STILL getting this error. I even cleaned out the git repo. I am using the same repo, but deleted everything and commited the delete before starting the registry up. I am a bit lostany thoughts/ideas? David
Re: NiFi docker container fails to start on VMWare host
It was because of this: https://stackoverflow.com/questions/72841549/container-fails-to-start-insufficient-memory-for-the-java-runtime-environment-t#:~:text=Possible%20reasons%3A%20The%20system%20is,physical%20memory%20or%20swap%20space The customer had installed docker 18.xx, and we have not had to specify the version before and this new NiFi 1.19.X version was put out with minimal testing on our servers (i.e. didn't check with older docker versions. Upgrading to docker 20.10.13 resolved the issue. Thanks! On Mon, Jan 30, 2023 at 11:52 PM Chris Sampson wrote: > Have you tried an earlier version of nifi in the same environment? > > Does the image for nifi 1.18.0 boot successfully, for example? > > The reason I'd check this is mainly that the apache/nifi convenience > docker image switched from java 8 to java 11 for nifi version 1.19.0+. I > think the way in which memory is allocated to the jvm within a docker > container changed between java 8 and 11, so it might be worth checking to > see whether an earlier version works. > > Is the VMWare environment 64 or 32bit as I think that can also affect how > the jvm allocates memory (based on a brief search for parts of your error > message - https://www.baeldung.com/jvm-compressed-oops)? > > > On Tue, 31 Jan 2023, 03:36 Cannon Palms, wrote: > >> Check the resources on the host. >> >> By default, the docker container will have all of the available memory on >> the host machine available to it. If you'd like to ensure that this >> available memory is at least X, you can use a memory reservation in your >> docker compose file: >> >> ``` >> ... >> mem_reservation: 4Gi >> ``` >> >> etc. >> >> You are correct, though, that this is not a Nifi-specific question. You >> may have more success on a docker-specific forum or platform. >> >> Best, >> Cannon >> >> >> On Mon, Jan 30, 2023, 8:27 PM David Early via users < >> users@nifi.apache.org> wrote: >> >>> Hi all, >>> >>> We are deploying several NiFi containers as part of a service for a >>> customer and we have run into an issue we have not seen before. >>> >>> I THINK the problem may be more of a problem with Java/Docker/VMWare >>> than NiFi per se, but I wanted to ask here in case someone has seen this >>> before. >>> >>> We recently upgraded a small system in Azure to 1.19.1 using the >>> "official" docker container. This worked just fine. The host was an Azure >>> VM running an older CentOS 7 image. There were no issues. >>> >>> For the current install, we used the same Docker image but it is on prem >>> with a customer who is using VMWare to create the hosts we are running on. >>> The hosts (3 of them) are 8 core/64G. >>> >>> The error we are getting is: >>> >>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >>>> file /opt/nifi/nifi-current/conf/bootstrap.conf >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >>>> file /opt/nifi/nifi-current/conf/bootstrap.conf >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >>>> file /opt/nifi/nifi-current/conf/nifi.properties >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >>>> file /opt/nifi/nifi-current/conf/nifi.properties >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >>>> file /opt/nifi/nifi-current/conf/nifi.properties >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >>>> file /opt/nifi/nifi-current/conf/nifi.properties >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >>>> file /opt/nifi/nifi-current/conf/nifi.properties >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >>>> file /opt/nifi/nifi-current/conf/nifi.properties >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >>>> file /opt/nifi/nifi-current/conf/nifi.properties >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >>>> file /home/nifi/.nifi-cli.nifi.properties >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >>>> file /home/nifi/.nifi-cli.nifi.properties >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >>>> file /home/nifi/.nifi-cli.nifi.properties >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing ta
NiFi docker container fails to start on VMWare host
Hi all, We are deploying several NiFi containers as part of a service for a customer and we have run into an issue we have not seen before. I THINK the problem may be more of a problem with Java/Docker/VMWare than NiFi per se, but I wanted to ask here in case someone has seen this before. We recently upgraded a small system in Azure to 1.19.1 using the "official" docker container. This worked just fine. The host was an Azure VM running an older CentOS 7 image. There were no issues. For the current install, we used the same Docker image but it is on prem with a customer who is using VMWare to create the hosts we are running on. The hosts (3 of them) are 8 core/64G. The error we are getting is: Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target file > /opt/nifi/nifi-current/conf/bootstrap.conf > Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target file > /opt/nifi/nifi-current/conf/bootstrap.conf > Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target file > /opt/nifi/nifi-current/conf/nifi.properties > Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target file > /opt/nifi/nifi-current/conf/nifi.properties > Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target file > /opt/nifi/nifi-current/conf/nifi.properties > Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target file > /opt/nifi/nifi-current/conf/nifi.properties > Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target file > /opt/nifi/nifi-current/conf/nifi.properties > Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target file > /opt/nifi/nifi-current/conf/nifi.properties > Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target file > /opt/nifi/nifi-current/conf/nifi.properties > Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target file > /home/nifi/.nifi-cli.nifi.properties > Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target file > /home/nifi/.nifi-cli.nifi.properties > Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target file > /home/nifi/.nifi-cli.nifi.properties > Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target file > /home/nifi/.nifi-cli.nifi.properties > Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target file > /home/nifi/.nifi-cli.nifi.properties > Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target file > /opt/nifi/nifi-current/conf/nifi.properties > Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target file > /opt/nifi/nifi-current/conf/nifi.properties > Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target file > /opt/nifi/nifi-current/conf/nifi.properties > Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target file > /opt/nifi/nifi-current/conf/nifi.properties > Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target file > /opt/nifi/nifi-current/conf/nifi.properties > Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target file > /opt/nifi/nifi-current/conf/nifi.properties > Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target file > /opt/nifi/nifi-current/conf/nifi.properties > Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target file > /opt/nifi/nifi-current/conf/nifi.properties > Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target file > /opt/nifi/nifi-current/conf/nifi.properties > Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target file > /opt/nifi/nifi-current/conf/nifi.properties > Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target file > /opt/nifi/nifi-current/conf/nifi.properties > Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target file > /opt/nifi/nifi-current/conf/nifi.properties > Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target file > /opt/nifi/nifi-current/conf/nifi.properties > Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target file > /opt/nifi/nifi-current/conf/nifi.properties > Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target file > /opt/nifi/nifi-current/conf/nifi.properties > Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target file > /opt/nifi/nifi-current/conf/nifi.properties > Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: > Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: Java home: > /opt/java/openjdk > Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: NiFi home: > /opt/nifi/nifi-current > Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: > Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: Bootstrap Config > File: /opt/nifi/nifi-current/conf/bootstrap.conf > Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: > Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: > [0.003s][warning][os,thread] Failed to start thread "GC Thread#0" - > pthread_create
Re: Issue with removal and re-add of a cluster node
Forgot my version: 1.16.3 Dave On Fri, Dec 9, 2022 at 11:22 AM David Early wrote: > Hi all, > > I have a major issue and am not sure what to do about it. > > We have a 3 node cluster. I was working on a one-off load for some data > we were doing out of sequence and it resulted in build-up of some flowfiles > in a queue. In order to prevent a backpressure situation, I cleared one of > the holding queues that had about 69k flow files. > > During the clear operation the node I was on (node 3 UI in this case) > returned and stated that the node was no longer part of the cluster. Not > clear why that happened. > > This, by itself, is not really an issue. Looking at the logs (at bottom > of this note), you can see theflowfile drop and immediate adjustment to the > node 3 to state of CONNECTING to the cluster. Subsequently, an error > occurred: "*Disconnecting node due to Failed to properly handle > Reconnection request due to > org.apache.nifi.controller.serialization.FlowSynchronizationException: > Failed to connect node to cluster because local flow controller partially > updated. Administrator should disconnect node and review flow for > corruption*". > > When I attempted to readd the node from the UI, it repeated this error. > > I compared users.xml and authroizations.xml on all three nodes, textually > the same and identical md5sum on all (issues with users.xml and > authorizations.xml were listed online as usual suspects). > > I then offloaded the node via the UI to make sure I didn't have anything > stuck in queue on node 3 and hoped it would allow the node to rejoin. > After offloading, I attempted to reconnect and what happened next gave me a > heart attack: Node 3 now showed as connected but in the UI (accessed via > node 1), ALL PROCESSORS WERE SHOWN AS STOPPED. > > A quick review showed that traffic was actually flowing (View status > history showed flowfiles moving, observing some of our observation queues > showed traffic on nodes 1 and 2). Removing node 3 from the cluster > restored the UI to show everything running, but adding it back in showed > everything as stopped. > > I tried to start some processors while node 3 was connected and while I > could start individual processors, I could not do a "global" start by right > clicking on canvas and trying "start". I set up a sample processor to > generate a file on all 3 nodes and it did generate a new flowfile on node > 3. All of that worked fine. > > We have 400+ processors that I would need to hand start and I am super > nervous about the cluster deciding to make node 3 the primary which would > affect some timed process that we are running on the primary node. As long > as I don't restart the http input feed, I COULD restart all the processors, > but this seems like the wrong process. > > Anyone have any idea what I did wrong and how to fix it? The errors show > in the log attached happened before any offloading, but I wondered if the > offloading caused part of this issue. > > Is there anything I can do to readd the node without having to restart all > the processors manually? > > Should I clean up the node and add it as a "new" node and let it > completely sync? > > Thanks for any insight! > > > Dave > > > --- > Log: > --- > *2022-12-08 22:26:20,706 INFO [Drop FlowFiles for Connection > 8b0ee741-0183-1000--68704c93] o.a.n.c.queue.SwappablePriorityQueue > Successfully dropped 69615 FlowFiles (35496003 bytes) from Connection with > ID 8b0ee741-0183-1000--68704c93 on behalf of u...@org.com > * > 2022-12-08 22:26:20,707 INFO [Process Cluster Protocol Request-29] > o.a.n.c.c.node.NodeClusterCoordinator Status of prod-stsc2-3:8443 changed > from NodeConnectionStatus[nodeId=prod-stsc2-3:8443, state=CONNECTED, > updateId=108] to NodeConnectionStatus[nodeId=prod-stsc2-3:8443, > state=CONNECTING, updateId=114] > 2022-12-08 22:26:20,707 INFO [Process Cluster Protocol Request-29] > o.a.n.c.p.impl.SocketProtocolListener Finished processing request > 070fe65c-4a77-41d0-9d7f-8f08ede6ac71 (type=NODE_STATUS_CHANGE, length=1217 > bytes) from prod-stsc2-1.internal.cloudapp.net in 10 seconds, 842 millis > 2022-12-08 22:26:20,750 INFO [Reconnect to Cluster] > o.a.nifi.controller.StandardFlowService Setting Flow Controller's Node ID: > prod-stsc2-3:8443 > 2022-12-08 22:26:20,751 INFO [Reconnect to Cluster] > o.a.n.c.s.VersionedFlowSynchronizer Synchronizing FlowController with > proposed flow: Controller Already Synchronized = true > 2022-12-08 22:26:21,298 INFO [NiFi Web Server-1481911] > o.a.c.f.imps.CuratorFrameworkImpl Starting > 2022-12-08 22:26:21,298 INFO [NiFi Web Server-1481911] > org.apache.zookeeper.ClientCnxnSocket jute.maxbuffer value is 4194304 Bytes > 2022-12-08 22:26:21,304 INFO [NiFi Web Server-1481911] > o.a.c.f.imps.CuratorFrameworkImpl Default schema > 2022-12-08 22:26:21,314 INFO [NiFi Web Server-1481911-EventThread] > o.a.c.f.state.ConnectionStateManager
Issue with removal and re-add of a cluster node
Hi all,
I have a major issue and am not sure what to do about it.
We have a 3 node cluster. I was working on a one-off load for some data we
were doing out of sequence and it resulted in build-up of some flowfiles in
a queue. In order to prevent a backpressure situation, I cleared one of
the holding queues that had about 69k flow files.
During the clear operation the node I was on (node 3 UI in this case)
returned and stated that the node was no longer part of the cluster. Not
clear why that happened.
This, by itself, is not really an issue. Looking at the logs (at bottom of
this note), you can see theflowfile drop and immediate adjustment to the
node 3 to state of CONNECTING to the cluster. Subsequently, an error
occurred: "*Disconnecting node due to Failed to properly handle
Reconnection request due to
org.apache.nifi.controller.serialization.FlowSynchronizationException:
Failed to connect node to cluster because local flow controller partially
updated. Administrator should disconnect node and review flow for
corruption*".
When I attempted to readd the node from the UI, it repeated this error.
I compared users.xml and authroizations.xml on all three nodes, textually
the same and identical md5sum on all (issues with users.xml and
authorizations.xml were listed online as usual suspects).
I then offloaded the node via the UI to make sure I didn't have anything
stuck in queue on node 3 and hoped it would allow the node to rejoin.
After offloading, I attempted to reconnect and what happened next gave me a
heart attack: Node 3 now showed as connected but in the UI (accessed via
node 1), ALL PROCESSORS WERE SHOWN AS STOPPED.
A quick review showed that traffic was actually flowing (View status
history showed flowfiles moving, observing some of our observation queues
showed traffic on nodes 1 and 2). Removing node 3 from the cluster
restored the UI to show everything running, but adding it back in showed
everything as stopped.
I tried to start some processors while node 3 was connected and while I
could start individual processors, I could not do a "global" start by right
clicking on canvas and trying "start". I set up a sample processor to
generate a file on all 3 nodes and it did generate a new flowfile on node
3. All of that worked fine.
We have 400+ processors that I would need to hand start and I am super
nervous about the cluster deciding to make node 3 the primary which would
affect some timed process that we are running on the primary node. As long
as I don't restart the http input feed, I COULD restart all the processors,
but this seems like the wrong process.
Anyone have any idea what I did wrong and how to fix it? The errors show
in the log attached happened before any offloading, but I wondered if the
offloading caused part of this issue.
Is there anything I can do to readd the node without having to restart all
the processors manually?
Should I clean up the node and add it as a "new" node and let it completely
sync?
Thanks for any insight!
Dave
---
Log:
---
*2022-12-08 22:26:20,706 INFO [Drop FlowFiles for Connection
8b0ee741-0183-1000--68704c93] o.a.n.c.queue.SwappablePriorityQueue
Successfully dropped 69615 FlowFiles (35496003 bytes) from Connection with
ID 8b0ee741-0183-1000--68704c93 on behalf of u...@org.com
*
2022-12-08 22:26:20,707 INFO [Process Cluster Protocol Request-29]
o.a.n.c.c.node.NodeClusterCoordinator Status of prod-stsc2-3:8443 changed
from NodeConnectionStatus[nodeId=prod-stsc2-3:8443, state=CONNECTED,
updateId=108] to NodeConnectionStatus[nodeId=prod-stsc2-3:8443,
state=CONNECTING, updateId=114]
2022-12-08 22:26:20,707 INFO [Process Cluster Protocol Request-29]
o.a.n.c.p.impl.SocketProtocolListener Finished processing request
070fe65c-4a77-41d0-9d7f-8f08ede6ac71 (type=NODE_STATUS_CHANGE, length=1217
bytes) from prod-stsc2-1.internal.cloudapp.net in 10 seconds, 842 millis
2022-12-08 22:26:20,750 INFO [Reconnect to Cluster]
o.a.nifi.controller.StandardFlowService Setting Flow Controller's Node ID:
prod-stsc2-3:8443
2022-12-08 22:26:20,751 INFO [Reconnect to Cluster]
o.a.n.c.s.VersionedFlowSynchronizer Synchronizing FlowController with
proposed flow: Controller Already Synchronized = true
2022-12-08 22:26:21,298 INFO [NiFi Web Server-1481911]
o.a.c.f.imps.CuratorFrameworkImpl Starting
2022-12-08 22:26:21,298 INFO [NiFi Web Server-1481911]
org.apache.zookeeper.ClientCnxnSocket jute.maxbuffer value is 4194304 Bytes
2022-12-08 22:26:21,304 INFO [NiFi Web Server-1481911]
o.a.c.f.imps.CuratorFrameworkImpl Default schema
2022-12-08 22:26:21,314 INFO [NiFi Web Server-1481911-EventThread]
o.a.c.f.state.ConnectionStateManager State change: CONNECTED
2022-12-08 22:26:21,322 INFO [NiFi Web Server-1481911-EventThread]
o.a.c.framework.imps.EnsembleTracker New config event received:
{server.1=prod-zkpr-1:2888:3888:participant;0.0.0.0:2181, version=0,
Re: nifi-api with a server secured with Microsoft AD
Mike and Shawn, thanks for the feedback, have not had a chance to try either, but appreciate your help. Will be trying the cert this week, will reach out to the AD managers about a more direct AD solution. Dave On Sat, Oct 29, 2022 at 7:10 PM Mike Thomsen wrote: > David, > > Another option you might want to explore is having AD generate client > certificates for your users. > > On Sat, Oct 29, 2022 at 12:01 PM Shawn Weeks > wrote: > > > > NiFi should always accept a cert at the rest api if you provide one. If > your using curl just add the “--key” and “--cert” and call whatever api url > your trying directly. You’ll need to make sure that the cert your using is > signed by the same local CA that NiFi is set to trust and that you’ve added > a user in NiFi that matches the common name on the cert or whatever regex > you set for “nifi.security.identity.mapping.value.pattern” > > > > Thanks > > Shawn > > > > > On Oct 28, 2022, at 3:55 PM, David Early via users < > users@nifi.apache.org> wrote: > > > > > > Hi all, > > > > > > We have a 3 node cluster secured with Microsort AD for the first time. > > > > > > I need access to the REST api. The nifi-api/access/token does not > work in this case. > > > > > > We did use a local CA for certificate generation on the servers. > > > > > > I am reading that it is possible to do certificate based auth to the > apiwe need this in a script (python) to run on a remote server which is > checking for old flowfiles that can get stuck in a few places. > > > > > > Can I use cert based API connection when using AD as the main > authentication/authorization for the ui? > > > > > > Anything special that needs to be done? I've just not used certs with > the api before, but we have used cert based site to site on other systems > and it works fine. Just not sure how to do it with nipyapi or just from > curl on the cli. > > > > > > David > > > -- David Early, Ph.D. david.ea...@grokstream.com 720-470-7460 Cell
nifi-api with a server secured with Microsoft AD
Hi all, We have a 3 node cluster secured with Microsort AD for the first time. I need access to the REST api. The nifi-api/access/token does not work in this case. We did use a local CA for certificate generation on the servers. I am reading that it is possible to do certificate based auth to the apiwe need this in a script (python) to run on a remote server which is checking for old flowfiles that can get stuck in a few places. Can I use cert based API connection when using AD as the main authentication/authorization for the ui? Anything special that needs to be done? I've just not used certs with the api before, but we have used cert based site to site on other systems and it works fine. Just not sure how to do it with nipyapi or just from curl on the cli. David
Issues with autoreload of truststore
All, We have 2 systems, one running 1.15.1 and one running 1.16.3. We have tried to use the nifi.security.autoreload.enabled=true feature and not had it work in the past and then suddenly yesterday on the 1.16.3 system it WORKED but we can't figure out why. In this case it WORKED when we accidentally copied in a truststore that had been set up as JKS and the config was asking for PKCS12, so it failed. So we tried again on the 1.15.1 system, and while we get the Scanner entries that appear to notice the change, nothing actually happens. I quickly went through the release notes and searched here but didn't find anything specific about changes to improve this. Any suggestions? We would LOVE to pick up a new truststore but at this point, we can't figure out how to kick off the change. log entry example of the scanner: Sep 9 16:36:46 prod-sts-1 nifi_1|bcf752078c86[2247]: 2022-09-09 16:36:46,644 INFO [Scanner-0] o.e.jetty.util.ssl.SslContextFactory x509=X509@6f0da1a2(1,h=[prod-sts-1, prod-sts-1.internal, prod-sts-1.com, sts-prod.company.net],a=[],w=[]) for Server@621d9971 [provider=null,keyStore=file:///opt/nifi/nifi-current/conf/ssl/nifi-keystore.jks,trustStore=file:///opt/nifi/nifi-current/conf/ssl/nifi-truststore.jks] Dave
Re: Crash on startup due to Output port issue
Benji,
Can you share your nifi.properties file?
This problem can show up if the disk fills up but depends on the
configuration.
Also, what does df -h show for your disks (assuming you are running on
Linux)?
Dave
On Sun, Jul 31, 2022, 2:21 PM Benji Benning wrote:
> Not really around the below error.
> Anything I can move to debug or trace in logback.xml to add some color
> around this?
>
> Thanks,
> Benji
>
> Sent from my iPhone
>
> On Jul 31, 2022, at 15:12, Joe Witt wrote:
>
>
> Benji
>
> Is there more with the logs like component uuids and such?
>
> Thanks
>
> On Sun, Jul 31, 2022 at 1:08 PM Benji Benning
> wrote:
>
>> Hi Joe,
>> Unfortunately it does indeed contain sensitive info.
>> Is there any debug info I can provide?
>>
>> Thanks,
>> Benji
>>
>> Sent from my iPhone
>>
>> On Jul 31, 2022, at 12:25, Joe Witt wrote:
>>
>>
>>
>> Can you share the flow or does it contain sensitve info?
>>
>> On Sun, Jul 31, 2022 at 10:01 AM BeNJ wrote:
>>
>>> Hi,
>>> I'm a little at a loss so I figured maybe someone here would have an
>>> idea of how to get out of this situation.
>>> My NiFI is crashing on startup with the following:
>>> --
>>> 2022-07-31 03:10:05,448 INFO [main] o.a.n.c.s.AffectedComponentSet
>>> Starting the following components: AffectedComponentSet[inputPorts=[],
>>> outputPorts=[], remoteInputPorts=[], remoteOutputPorts=[], processors=[],
>>> controllerServices=[], reportingTasks=[]]
>>> 2022-07-31 03:10:05,452 WARN [main]
>>> org.eclipse.jetty.webapp.WebAppContext Failed startup of context
>>> o.e.j.w.WebAppContext@3ed03652
>>> {nifi-api,/nifi-api,file:///opt/nifi/nifi-current/work/jetty/nifi-web-api-1.16.0.war/webapp/,UNAVAILABLE}{./work/nar/extensions/nifi-server-nar-1.16.0.nar-unpacked/NAR-INF/bundled-dependencies/nifi-web-api-1.16.0.war}
>>> org.apache.nifi.controller.serialization.FlowSynchronizationException:
>>> java.lang.IllegalStateException: Cannot add Connection to Process Group
>>> because source is an Output Port that does not belong to a child Process
>>> Group
>>> --
>>> I only restarted because out of nowhere i started seeing the following
>>> errors repeating:
>>> Processors cannot write to Content Repo: Unable to write flowfile
>>> content to content repository container default due to archive file size
>>> constraints; waiting for archive cleanup
>>>
>>> To resolve this issue, someone said online to set
>>> nifi.content.repository.archive.enabled to false. I restarted and nifi
>>> wouldn't come up again.
>>>
>>> I've tried reverting the flow.xml.gz to the oldest one in the archive
>>> folder (was from 7pm yesterday), no luck.
>>> it's a large config (1000+ active processors).
>>> I would try to disable / delete the offending output port, but I don't
>>> see how to find it from the logs.
>>>
>>> Any direction would be greatly appreciated.
>>>
>>> Thanks,
>>> Benji
>>>
>>
