Re: question about ConsumeKafka metrics for incoming data

t downstream, you can look at "bytes out". > The value should be close to how many bytes have been consumed from Kafka. > I'm not familiar with the Kafka processors, but I assume the messages are > encrypted/compressed. > > On Thu, Oct 18, 2018 at 11:25 AM Dominique De

question about ConsumeKafka metrics for incoming data

Hi, While running a ConsumeKafka processor, I noticed there are metrics for incoming data, but there was no value (like "bytes in") provided in the UI. And as far as I have digged into the ConsumeKafka processor, I didn't found any code related to feeding the metrics related to the incoming

how to organise processor or PG in the Nifi canvas ?

Hi, Is there a way to re-organize automatically the processors or PG inside the Nifi canvas ? I create PG through REST api. I set originX and originY for all PG I create, but the created PG could be on top of another PG (for example, previously existing) and the first one could be hiding the

Re: problem with instanciating NEW (custom) processors

On Fri, Sep 14, 2018 at 6:59 AM Dominique De Vito > wrote: > >> Hi, >> >> I have defined 5 new (custom) processors (derivated respectively from >> existing ones - ConsumeKafka, GetSFTP, GetFile, ListenTCP an ListFile) into >> 2 NAR (1 NAR with 1 processor, t

Re: problem with instanciating NEW (custom) processors

quot; Anyway, thanks for the idea. Dominique Le ven. 14 sept. 2018 à 13:42, Mike Thomsen a écrit : > Did you change their names when you created your new versions? > On Fri, Sep 14, 2018 at 6:59 AM Dominique De Vito > wrote: > >> Hi, >> >> I have defined 5 new

problem with instanciating NEW (custom) processors

Hi, I have defined 5 new (custom) processors (derivated respectively from existing ones - ConsumeKafka, GetSFTP, GetFile, ListenTCP an ListFile) into 2 NAR (1 NAR with 1 processor, the other processors are into the other NAR). After copying the NARs into /lib, and after restarting Nifi, only 3

Re: question about ListFile processor

hat node. > > When the directory is remote then it is intended to be run on primary > node and the state is stored cluster wide so that if primary changes > then another node can take over. > > -Bryan > > > On Wed, Sep 12, 2018 at 5:55 AM, Dominique De Vito > wrote: > &

question about ListFile processor

Hi, The property "Input Directory Location" of ListFile has two possible values: "local" or "remote". According to the doc, it "*specifies where the Input Directory is located. This is used to determine whether state should be stored locally or across the cluster*." The same doc [1] states

Re: problem with Nifi / Atlas integration - has anyone some experience with this integration ?

xpected > KAFKA_HOME/bin/kafka-consumer-groups.sh --bootstrap-server server:port > --describe --group atlas > GROUP TOPIC > PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG OWNER > atlas ATLAS_HOOK 0 > 2

problem with Nifi / Atlas integration - has anyone some experience with this integration ?

Hi, I have defined a simple pipeline in Nifi: GetFile => some processor doing a dummy transformation => PublishInKafka ...with Atlas integration for lineage purposes Versions: -- Atlas 0.8.0 (Stack : HDP 2.6.4) -- Nifi 1.5.0 and I have put some (dummy) file into the input

Re: Processor example with dynamically fetched values for attributs

ou > would create a new PropertDescriptor for the given property using the > latest list, but there is nothing built in to help with this. > > It also doesn't work well for documentation which is auto-generated > when the application starts and would have no way of knowing what the >

Processor example with dynamically fetched values for attributs

Hi, I try to define a Processor such as for a given attribut: * the possible values are dynamically fetched from a REST WS (from time to time) * the UI offers to select one of these values from a drop-down list. Is there some processor implementing such a pattern? If so, then, I may take it as

Re: Best Pratices for implementing a source catalog for Nifi

hose will work with pre-defined sources and destinations) to users in a > custom portal, ask user specific configurations in the UI (custom) based on > template selection and integrate this UI with NiFi using REST. > > > > Thanks, > > Ravi Papisetti > > > >

Best Pratices for implementing a source catalog for Nifi

Hi, I just wondering what are the Best Pratices for implementing a source catalog for Nifi. I mean the following: * Nifi is able to catch data from various sources, and that's just fine (there are plenty of processors!) * In a more constrained world, I would like to limit the sources a Nifi

Re: question about "NiFi lineage can now be exported to Apache Atlas"

. > > Does that make sense? > > Thanks > -Mark > > > > On Jan 23, 2018, at 7:15 AM, Dominique De Vito <ddv36...@gmail.com> > wrote: > > > > Hi, > > > > AFAIK lineage occurs at FlowFile level. > > That is (AFAIU) each FlowFile could have it

question about "NiFi lineage can now be exported to Apache Atlas"

Hi, AFAIK lineage occurs at FlowFile level. That is (AFAIU) each FlowFile could have it its own lineage. So, if Nifi is reading a file as input, with 1 000 records, is Nifi going to send to Atlas 1 000 lineages? If yes, does Nifi send these 1 000 lineages (to Atlas) in one call in a batch way?