Re: Monitoring NiFi Root Level Controller Services

[jgunvaldson]

Hi Daniel,

And yes I am “John” (Joe was correct) - I have some experience with Nipyapi and 
yes, will probably be using it - I was wondering about a 3rd party (not 
DataDog) monitoring service (designed) for working with NiFi as recommendations 
from the group, but certainly - writing our own monitors to use API is probably 
what we will do

Best Regards
John




> On Feb 1, 2021, at 10:48 PM, Daniel Chaffelson  wrote:
> 
> Hi John, the python client nipyapi already has convenience methods in 
> canvas.py for this, you are looking for a component status of Running.
> 
> On Tue, 2 Feb 2021, 01:18 jgunvaldson,  > wrote:
> Wiring up a rest client to make a rest request to the NIFI API, seems the 
> following three are possible candidates:
> 
> GET /controller-services/{id}  Gets a controller 
> service
> 
> GET /controller-services/{id}/referencesGets a controller service
> 
> GET /controller-services/{id}/state Gets the state for a 
> controller service
> 
> I’ll do some initial tests with Postman to see what is returned in response 
> json payload (and if 200 means up and running, or just a successful API 
> request)
> 
> Sounds good
> John
> 
> 
> 
>> On Feb 1, 2021, at 4:46 PM, Joe Witt > > wrote:
>> 
>> John,
>> 
>> You're using a vendor distribution of NiFi.  You should contact the vendor.
>> 
>> You can certainly monitor the state of a controller service via the REST 
>> API.  They should either be enabled or not enabled.
>> 
>> Thanks
>> 
>> On Mon, Feb 1, 2021 at 5:39 PM jgunvaldson > > wrote:
>> Hi All,
>> 
>> Root level (canvas) Controller Services - We tend to setup several Root 
>> level Controller services for developers that are typically 
>> DBCPConnectionPools and DistributedMapCacheClientService and maybe a few 
>> other. MOST importantly, these controller services cannot be down and cannot 
>> be disabled - must remain Enabled at all times.
>> 
>> We have now had a few outages where upon examination a Controller Service 
>> has encountered “something” that caused it to be Disabled or Down.
>> 
>> Is there a standard practice we can use to “Monitor” the controller services 
>> and ensure we get alerted if any one of them goes into a Disable state?
>> 
>> What do folks generally think is a good monitoring practice?
>> 
>> We are on 
>> 
>> HDF Version 3.4.1.1.
>> 
>> Powered by Apache NiFi Version 1.9.0
>> 1.9.0.3.4.1.1-4 built 05/01/2019 02:15:30 UTC
>> Tagged nifi-1.9.0-RC2
>> From 7410fa4 on branch UNKNOWN
>> 
>> 
> 

Monitoring big directory tree

[Tomislav Novosel]

Hi guys,

I have following situation:

There is SMB mounted folder on one Nifi worker and it has many subfolders with 
subfolders (the depth of nesting is not known in advance).
If new files in that directory tree appears or file is moved or old file is 
copied/moved, modification timestamp changes. It is achieved
with some other tools, configs etc.

What is the best way to list new/updated/new-old files with NiFi if we take 
into account there are enormous number of subfolders and files in them (let's 
say milions).

In case of ListFile and using 'Tracking Entities' I am concerned about the 
following:


  *   How big is the I/O if ListFile constantly checks the directory tree and 
all files?
It can be CRON based not to do it all the time, but if it is not, how ListFile 
is doing that in the background?
  *   How big is the cahce of listed entities, what is stored in fact in the 
cache, just metadata or?
  *   What if cahce is not persisted and NiFi restart occurs? Will cache be 
incosistent?
In case it is persisted, what if restart occurs in the moment when ListFile is 
checking new/old entities and their size, name etc?
What is the interval of persisting the cache, is it related to snapshots NiFi 
takes in configured intervals?

What is the best and the most efficient way to do this? Maybe some extra tools 
or engines to use for finding difference and to
persist last known state, like elastic, some DB maybe?
Or to construct list of paths which need to be fecthed using some python 
sctipts?

The constarint here is shared(mounted) folders and even if modification date is 
changed for every new/updated file,
how to efficiently monitor big directory tree or how to efficiently trigger 
ListFile (NiFi flow) to fetch new/old-new files?

In case of 'Tracking entities', maybe having separated standalone NiFi instance 
on separated server with configured CacheServer
to serve as cache is not bad idea?

Thanks in advance,

Tom

Re: Week of year - query record processor

[Mark Payne]

You should be able to use something like:

${field.value:toDate(‘-MM-dd HH:mm:ss’):format(‘w’)}

Thanks
-Mark

> On Feb 1, 2021, at 9:13 PM, KhajaAsmath Mohammed  
> wrote:
> 
> Hi,
> 
> Is there a way to get weekofyear from the query record processor. Any syntax 
> would really help.
> 
> Time stamp: 2021-02-01 12:12:12
> 
> Thanks,
> Khaja
> 
> Sent from my iPhone