Hi Ravi,
To distribute QueryDatabaseTable workload, I'd suggest using
GenerateTableFetch instead. Because it can generate SQLs to query
updated records. And those SQL FlowFiles can be distributed among NiFi
nodes by RPG.
-
Following lines are just to share my thoughts on the topic for
possib
Eventually I find a way to make it work. The "init" method is triggered when
the service instance is created before the property is configured/ready. I
can't do data loading there.
I need to prepare another API to lazily load data from database, before any
call of "getXXX(String key)". As it is
Thanks, Matt. It works perfect!
Another question, I try to put the logic into a new controller service, as I
want to load data from database and cache in service. So I write java code with
api and bundle project and build in mvn, similar with NiFi standard service. In
this enrichment service,
Kui,
The getControllerService() method requires a controller service (CS)
identifier, not the name (because names are not necessarily unique).
To get the CS by name, you have to get the list of all CSs and match
on the name. I have an example in Groovy on my blog [1], but here is a
similar one por
Thanks Joe for confirmation.
I think, problem gets interesting when we start querying data from data bases
such as Oracle(QueryDatabaseTable) and Hbase (GetHbase). Are these also
expected to configure with “PrimaryNode”? RPG doesn’t seem to be recommended to
distribute huge content across clust
Ravi
The pattern of List -> -> Fetch -> Merge -> Put is common
and is the correct way to use it.
ListFile can often be used across the cluster all at once and work out
fine since each node is often accessing a unique resource like a local
file system. However, in this case you're pointing them
Hi,
We are creating processgroups to transfer data from File system to HDFS/HBASE
(separate process groups for each destination).
Simple Example: FileSystem to HDFS process group: We have ListFile, FetchFile
and PutHDFS process to transfer data from File system to HDFS. When “ListFile”
process
That was an awesome answer/clarification Raman! Thanks
On Thu, Dec 21, 2017 at 10:21 AM, Ramaninder Singh Jhajj
wrote:
> Hi Sally,
>
> You are facing issue with 10K files because that is the default value for
> backpressure in Queue. The queue will be full with 10K files and will
> initiate the
Hi Sally,
You are facing issue with 10K files because that is the default value for
backpressure in Queue. The queue will be full with 10K files and will
initiate the backpressure and your MergeContent processor never receive the
number of files you are expecting. You will need to change a couple
You can use MergeContent to merge more than 10,000 flowfiles, but you may
experience slower performance with very large numbers of files in a single
merge. The recommended configuration is to use two MergeContent processors
in sequence. The first MergeContent would merging groups of individual
fl
I have 1400 flowFile with same name and i want to enroll them into one file,
i have read that i can use mergeContent processor for this purpose but when
the amount of flowfile is more than 1 it can't combine them in one file,
what should i change to make this processor work properly? here is my
Hi,
I try to use javascript in processor to do some enrichment and transformation,
but i don't know how to concat the service identifier.
As it is ExecutorScript processor, I can't add a service reference in
properties, I can only add service in current Group (I put logic in one Process
Grou
12 matches
Mail list logo
