Re: CI/CD for custom processor in aws nifi cluster
Actually, I forgot auto-loading is in 1.9.0 [1], but even without that you could do the same thing and instead write it to your custom_lib, and then you'd have to call ExecuteScript to call "bin/nifi.sh restart". With auto-loading then no restart is required. [1] https://issues.apache.org/jira/browse/NIFI-5673 On Mon, Nov 18, 2019 at 12:09 PM sanjeet rath wrote: > > Thanks Bryan for your quick response. > we are using nifi 1.8 version.Could you please suggest something regards to > 1.8 version. > > Regards, > Sanjeet > > On Mon, 18 Nov, 2019, 10:21 PM Bryan Bende, wrote: >> >> If you are using NiFi 1.9.x or newer, then there is a new feature to >> auto-load new NARs in a directory specified by >> nifi.nar.library.autoload.directory in nifi.properties. >> >> You could make part of your NiFi flow run ListS3 -> FetchS3 -> PutFile >> into the auto-load directory, then NiFi will update itself and you >> won't need to restart. >> >> You'll want to make List3 run on all nodes since each node needs to >> get its own copy of the NAR. >> >> On Mon, Nov 18, 2019 at 11:42 AM sanjeet rath wrote: >> > >> > >> > Hi Team, >> > >> > I have a requirement of building CI/CD pipeline using jenkin & concourse >> > to build and deploy custom nars in AWS nifi cluster. >> > The CI part is completed successfully , but in the CD while doing the >> > deployment using concourse pipeline, we have a problem , we can not ssh >> > to port 22 of the nifi cluster nodes due to restrictions. >> > So we planed to copy the nar files from artifactory(jfrog) to s3 bucket & >> > from S3 bucket we will copy it to custom_lib folder of the nodes of aws >> > nifi cluster. >> > we have another restriction to avoid use of cron tab to (run on each niffi >> > cluster nodes )to pull it from s3 . >> > >> > Could you please suggest , how to copy files from s3 to cutom_lib folder >> > of nifi cluster and restart nifi service using jenkin and concourse >> > >> > Thanks >> > -- >> > Sanjeet Kumar Rath, >> > mob- +91 8777577470 >> >
Re: CI/CD for custom processor in aws nifi cluster
Thanks Bryan for your quick response. we are using nifi 1.8 version.Could you please suggest something regards to 1.8 version. Regards, Sanjeet On Mon, 18 Nov, 2019, 10:21 PM Bryan Bende, wrote: > If you are using NiFi 1.9.x or newer, then there is a new feature to > auto-load new NARs in a directory specified by > nifi.nar.library.autoload.directory in nifi.properties. > > You could make part of your NiFi flow run ListS3 -> FetchS3 -> PutFile > into the auto-load directory, then NiFi will update itself and you > won't need to restart. > > You'll want to make List3 run on all nodes since each node needs to > get its own copy of the NAR. > > On Mon, Nov 18, 2019 at 11:42 AM sanjeet rath > wrote: > > > > > > Hi Team, > > > > I have a requirement of building CI/CD pipeline using jenkin & concourse > to build and deploy custom nars in AWS nifi cluster. > > The CI part is completed successfully , but in the CD while doing the > deployment using concourse pipeline, we have a problem , we can not ssh to > port 22 of the nifi cluster nodes due to restrictions. > > So we planed to copy the nar files from artifactory(jfrog) to s3 bucket > & from S3 bucket we will copy it to custom_lib folder of the nodes of aws > nifi cluster. > > we have another restriction to avoid use of cron tab to (run on each > niffi cluster nodes )to pull it from s3 . > > > > Could you please suggest , how to copy files from s3 to cutom_lib folder > of nifi cluster and restart nifi service using jenkin and concourse > > > > Thanks > > -- > > Sanjeet Kumar Rath, > > mob- +91 8777577470 > > >
Re: CI/CD for custom processor in aws nifi cluster
If you are using NiFi 1.9.x or newer, then there is a new feature to auto-load new NARs in a directory specified by nifi.nar.library.autoload.directory in nifi.properties. You could make part of your NiFi flow run ListS3 -> FetchS3 -> PutFile into the auto-load directory, then NiFi will update itself and you won't need to restart. You'll want to make List3 run on all nodes since each node needs to get its own copy of the NAR. On Mon, Nov 18, 2019 at 11:42 AM sanjeet rath wrote: > > > Hi Team, > > I have a requirement of building CI/CD pipeline using jenkin & concourse to > build and deploy custom nars in AWS nifi cluster. > The CI part is completed successfully , but in the CD while doing the > deployment using concourse pipeline, we have a problem , we can not ssh to > port 22 of the nifi cluster nodes due to restrictions. > So we planed to copy the nar files from artifactory(jfrog) to s3 bucket & > from S3 bucket we will copy it to custom_lib folder of the nodes of aws nifi > cluster. > we have another restriction to avoid use of cron tab to (run on each niffi > cluster nodes )to pull it from s3 . > > Could you please suggest , how to copy files from s3 to cutom_lib folder of > nifi cluster and restart nifi service using jenkin and concourse > > Thanks > -- > Sanjeet Kumar Rath, > mob- +91 8777577470 >
CI/CD for custom processor in aws nifi cluster
Hi Team, I have a requirement of building CI/CD pipeline using jenkin & concourse to build and deploy custom nars in AWS nifi cluster. The CI part is completed successfully , but in the CD while doing the deployment using concourse pipeline, we have a problem , we can not ssh to port 22 of the nifi cluster nodes due to restrictions. So we planed to copy the nar files from artifactory(jfrog) to s3 bucket & from S3 bucket we will copy it to custom_lib folder of the nodes of aws nifi cluster. we have another restriction to avoid use of cron tab to (run on each niffi cluster nodes )to pull it from s3 . Could you please suggest , how to copy files from s3 to cutom_lib folder of nifi cluster and restart nifi service using jenkin and concourse Thanks -- Sanjeet Kumar Rath, mob- +91 8777577470
