Re: Bug/Unexpected behavior in ConvertJSONToSQL for boolean attributes

merged to master. Thanks for the report Jennifer and the super fast PR Koji. On Tue, Jan 2, 2018 at 7:23 PM, Koji Kawamura wrote: > Submitted a PR to fix the issue. > https://issues.apache.org/jira/browse/NIFI-4729 > https://github.com/apache/nifi/pull/2366 > > I

Re: use record reader to read text file and get line counts

Hello If you want to aggregate data by some filename/pattern use MergeContent and its correlation attribute feature. Once you've done the merging then use SplitText technique to get total line count. Thanks Joe On Tue, Jan 2, 2018 at 11:16 AM, tzhu wrote: > Hi Matt, >

Re: Nifi: how to combile 14 000 flowfile in one file?

That was an awesome answer/clarification Raman! Thanks On Thu, Dec 21, 2017 at 10:21 AM, Ramaninder Singh Jhajj wrote: > Hi Sally, > > You are facing issue with 10K files because that is the default value for > backpressure in Queue. The queue will be full with 10K

Re: Clarification on load distribution on NiFi cluster

Ravi The pattern of List -> -> Fetch -> Merge -> Put is common and is the correct way to use it. ListFile can often be used across the cluster all at once and work out fine since each node is often accessing a unique resource like a local file system. However, in this case you're pointing them

RE: [EXT] Re: NiFi Clustering

karthik Terminology wise you cannot share a repository across nodes. So that is a big con :) But if you mean can i share something like a network mounted drive by giving each node its own mount pointa for its repos then sure that works fine. In such a case you need to have strong iops

RE: Getting 404 from Nifi UI on new installation

hmm..ill have to get to a computer to compare logs. nothing obvious to me there yet. do your access logs show the requests? please share those too. On Jan 3, 2018 6:41 PM, "Bob Mroczka" wrote: > The URL I’m trying to access is listed in the app-log, third line

RE: Getting 404 from Nifi UI on new installation

can you show what url you are trying to access? On Jan 3, 2018 6:28 PM, "Bob Mroczka" wrote: > > > HCSC Company Disclaimer > > The information contained in this communication is confidential, private, > proprietary, or otherwise privileged and is intended only for the

RE: Getting 404 from Nifi UI on new installation

ok yeah the user log is what i was curious about. that is interesting. can you please run bin/nifi.sh dump then share the bootstrap.log. one log at a time we will get there On Jan 3, 2018 6:57 PM, "Bob Mroczka" wrote: > The only log files I see being written to

Re: Getting 404 from Nifi UI on new installation

hello please share the contents of the nifi app log. might need to creat a jira and attach or something similar. thanks joe On Jan 3, 2018 6:16 PM, "Bob Mroczka" wrote: > I’ve installed a standalone Nifi 1.4.0 instance on a Windows 2012 R2 > server. The instance

Re: Getting 404 from Nifi UI on new installation

Bob, There should be more in the app log after that controller initialization line and of course there should be things in the user log for when you're attempting access. Looking at the stack dump I'm thinking there is an issue with archive cleanup/management. It is either stuck on a native

Re: Getting 404 from Nifi UI on new installation

; doing this, so everything seems okay... I think the first step is to add in > some logging to at least make it obvious > what is happening. Then we can look at ways to make it faster on startup. > > Thanks > -Mark > > > [1] https://issues.apache.org/jira/browse/NIFI-473

Re: NiFi and Mongo

Kelsey Have you looked at JSONTreeReader? thanks joe On Thu, Jun 21, 2018, 5:00 AM Kelsey RIDER wrote: > OK, thanks for the heads-up! > > > > If I could make another suggestion: could the JSONPathReader be made a > little more dynamic? Currently you have to specify every single field… > > > >

Re: Question on ingesting HDFS batches

Sudhindra The current ListFile processor scans through the configured directory including any subdirectories and looks for files. It does this by generating a listing, comparing it to what it has seen already (largely based on mod time) then sending out resulting listings. These can be sent to a

Re: nifi-hive-nar-1.7.1.nar will not load

Geoff Dont think it is a known issue. Many of us tested the build/startup/etc.. Can you share the log output? Thanks On Thu, Jul 26, 2018 at 2:08 PM, geoff.craig wrote: > Hello, > > I did a clean install of NiFi 1.7.1 and nifi-hive-nar-1.7.1.nar will not > load. It throws a Java error. I

Re: Large JSON File Best Practice Question

ben are you familiar with the record readers, writers, and associated processors? i suspect if you make a record writer for your custom format at the end of the flow chain youll get great performance and control. thanks On Fri, Aug 10, 2018, 4:27 PM Benjamin Janssen wrote: > All, I'm seeking

Re: NiFi 1.6.0 cluster stability with Site-to-Site

; > [1] - https://issues.apache.org/jira/browse/NIFI-4598 > > -- Mike > > > On Fri, Aug 10, 2018 at 4:43 PM Joe Witt wrote: >> >> Joe G >> >> I do recall there were some fixes and improvements related to >> clustering performance/thread pooling/

Re:

Curtis Now that there is also a PR for this I'll comment directly there as well to the specifics of the PR. In reviewing the discussion here.. There is consensus that enabling the pattern of REST API interaction you need for your case is a valuable capability. However, we have not achieved

Re: NiFi 1.6.0 cluster stability with Site-to-Site

Joe G I do recall there were some fixes and improvements related to clustering performance/thread pooling/ as it relates to site to site. I dont recall precisely which version they went into but i'd strongly recommend trying the latest release if you're able. Thanks On Fri, Aug 10, 2018 at 4:13

Re: Get all Processors

Karthik I believe that call is/was very expensive on the server side. You might want to experiment with the latest release of NiFi against the same flow configuration. From conversations I have had I feel like this is an addressed issue though admittedly i'm not sure which JIRA would address it

Re: NiFi protobuf processor

James, I'd recommend you create or edit an existing JIRA with more of the details of the idea and how it would work. Use that to gauge whether there is enough interest perhaps before you go through the organization work of getting it released. It not leveraging the record reader/writer approach

Re: AVRO is the only output format with ExecuteSQL

i think we just need to make an ExecuteSqlRecord processor. thanks On Tue, Aug 7, 2018, 8:41 AM Mike Thomsen wrote: > My guess is that it is due to the fact that Avro is the only record type > that can match sql pretty closely feature to feature on data types. > On Tue, Aug 7, 2018 at 8:33 AM

Re: Large JSON File Best Practice Question

t to operate on the data in a manner that is > non-destructive to unanticipated fields. Is that a blocker for using the > RecordReader stuff? > > On Fri, Aug 10, 2018 at 4:30 PM Joe Witt wrote: >> >> ben >> >> are you familiar with the record readers, writers, and

Re: ConvertAvroSchema with unmapped fields

i'd also recommend avoiding that processor and instead moving to use the record oriented processors, readers, and writers. On Thu, Aug 23, 2018 at 1:28 PM Andy LoPresto wrote: > > Hi Dave, > > Can you provide the two schemas (redact anything necessary). There is a way > to specify an “optional”

Re: MergeContent prematurely binning flow files?

Tim, This processor is powerful and its configurations very specific. That is a fancy way of saying this beast is complicated. First, can you highlight which version of NiFi you're using? Lets look at your settings that would cause a group of items to get kicked out as a merge result:

Re: MergeContent prematurely binning flow files?

to a maximum as needed to constrain memory usage) > When no new flow files with a matching attribute value have come in for a > configurable duration (e.g. 5 minutes), merge all of the bin’s contents > together and move it on to the next processor. > > > Is there a better way to do

Re: Top right menu won't open after upgrade

Hello Sounds like possibly a caching issue. Try doing a hard cache clear/refresh in your web browser. Thanks On Mon, Jul 16, 2018 at 2:20 PM, Jean-Sebastien Vachon wrote: > Hi all, > > > > I just upgraded my 1.4.1 cluster to 1.7.1 and discovered that the top right > menu is not opening when

Re: Unable to create contentclaim

the error shows you are out of disk space which explains both problems. take a look at the best practices install guide to be sure your install is ideal for your environment. thanks On Thu, Jul 19, 2018, 4:37 PM Jones, Patrick L. wrote: > Greetings, > > > > I’m getting an

Re: Nifi RecordReader for Json?

Wow! My apologies for the really bad response I gave linking to the same article you mentioned. I should be more careful when reading/responding on the phone! Thanks On Thu, Aug 30, 2018 at 11:01 AM Matt Burgess wrote: > > V, > > Currently NiFi does not support specifying a schema in

Re: Demuxing NiFi logs

James, One log per process group is not currently supported. Through things like the message diagnostic context it might be pretty doable. That said, the logs aren't really meant to be the true historical record of value. The data provenance entries are. Can you share more about what you're

Re: Demuxing NiFi logs

hugely distracting. > > On Fri, 7 Sep 2018, 19:52 Joe Witt, wrote: >> >> James, >> >> One log per process group is not currently supported. Through things >> like the message diagnostic context it might be pretty doable. >> >> That said, the log

Re: RPG S2S Error

you can allow for larger backlogs by increasing the backpressure thresholds OR you can add additional nodes OR you can expire data. The whole point of the backpressure and pressure release features are to let you be in control of how many resources are dedicated to buffering data. However, in

Re: the question about nifi cache auto clear default(5 min)

Hello Please post to just one mailing list. This is a good question for users. Presently the 5 min window isn't configurable. However, on each component we do have a status history you can access that is retained longer (24 hours by default) and is configurable [1] Thanks [1]

Re: Multiple versions of .... exist.

Nice find and thanks for the contrib/fix! Already merged. There are plenty of things in already for a release to be worthwhile. Will have the discussion on dev. Thanks On Mon, Mar 12, 2018 at 12:12 PM, Jorge Machado wrote: > Yes, > > I opened a PR:

Re: Feature Request: Lock Processors onto the UI

hat about an undo feature for the UI? I often move > or change a processor and would really LOVE to be able to click undo. > > Thanks, > > Kevin > > -Original Message- > From: Joe Witt [mailto:joe.w...@gmail.com] > Sent: Wednesday, March 14, 2018 11:27 AM > To: users

Re: Feature Request: Lock Processors onto the UI

Kevin, We have talked about this a bit in the past. If you leverage versioned flows and you accidentally move or even make changes to the flow you could simply rollback the version to get in sync again. We could also add a way to enforce a read-only sort of view mode but there has never been

Re: Next Release

Jorge We dont generally publish a roadmap as they're really difficult to have any real detail/stability behind them. What we do is generally release couple months and we've started a discussion to put out Apache NiFi 1.6.0 soon with the release candidate/vote process starting ideally next week.

Re: Next Release

st wondering if its going to make it into the release. > > Thanks > > > Kiran > > > > ---- Original Message > On 14 Mar 2018, 15:00, Joe Witt < joe.w...@gmail.com> wrote: > > > Jorge > > We dont generally publish a roadmap as they're really difficul

Re: NiFi 1.6

dan It is presently working through the release candidate vote process. As it stands now it could be out tomorrow. Please do help by reviewing the rc if you have time. If you have questions on how to do it just let us know and we can help. thanks joe On Thu, Apr 5, 2018, 1:46 PM dan young

Re: Clustering Questions

Jon, Node Failure: You have to care about two things generally speaking. First is the flow execution and second is data in-flight For flow execution nifi clustering will take care of re-assigning the primary node and cluster coordinator as needed. For data we do not at present offer

Re: NiFi cluster with DistributedMapCacheServer/Client

James, You have it right about the proper solution path.. I think we have a Redis one in there now too that might be interesting (not in 1.3.0 perhaps but..). We offered a simple out of the box one early and to ensure the interfaces are right. Since then the community has popped up some

Re: Error in Jython environment

Independent of NiFi have you successfully used any version of Jython to make that call? With the NiFi 1.x line that processor has been updated a little bit and in particular i think we have Jython 2.7.1 (off memory so could be totally nonsense) where as before we had 2.7.0 or something older...

[ANNOUNCE] Apache NiFi 1.6.0 release

Hello The Apache NiFi team would like to announce the release of Apache NiFi 1.6.0. Apache NiFi is an easy to use, powerful, and reliable system to process and distribute data. Apache NiFi was made for dataflow. It supports highly configurable directed graphs of data routing, transformation,

Re: Best way to drop data based on its content

Laurens, The recommended model in terms of capability/performance/flexibility would be to use the Record Readers/Writers and associated processors as Andy was hinting at. The approach you took was the simplest and will perform fine up to say a few thousands per second. If you need faster then

Re: how to edit queue content

Scott Yep definitely something we've talked about [1]. We've not pursued it directly as of yet since it is indeed a queue and we're just letting you peak into it. We dont have facilities built in to really alter the queue in a particular position. Also, the complexity comes in when folks want

Re: GG kafka topic in avro format to NiFi

gt; consumer. I didn't want to create a properties file and couldn't see how to >>> set schema.name otherwise. >>> >>> Regarding the error information. I saw the error displayed at the >>> ConsumeKafka processor, from the UI. I will look for log files, and if &g

Re: GG kafka topic in avro format to NiFi

Can you share a template of your process group? Do the messages in Kafka have the Avro schema included in them or are they just the data portion of the record? On Wed, Mar 21, 2018 at 9:16 PM, Colin Williams wrote: > I have an avro avsc file for a table with a

Re: GG kafka topic in avro format to NiFi

: > Hi Joe, > > I don't believe the Avro schema included, and expect they are the data > portion... I think that's why I need to use the avsc file mentioned above... > > On Wed, Mar 21, 2018 at 6:19 PM, Joe Witt <joe.w...@gmail.com> wrote: >> >> Can you share

Re: How do I logout of NiFi UI

Vijay What mechanism are you using to login to NiFi? Thanks On Fri, Oct 12, 2018 at 10:37 AM Vijay Chhipa wrote: > > Hello > > I can't find the 'logout' link on the canvas or under my profile name or in > the hamburger menu. > Whats the recommended way to logout of Apache NiFi > > Can you

Re: NiFi 1.8 release date?

Dan There is a RC being prepared at this time for it. We'll start the vote within the next day or two it looks like. Thanks On Tue, Oct 16, 2018 at 6:03 PM dan young wrote: > > Any ideas on when 1.8 might be dropping? > > Regards, > > Dano >

Re: Cluster Warnings

the logs show the fourth node is the slowest by far in all cases. possibly a dns or other related issue? but def focus on that node as the outlier and presuming nifi config is identical it suggest system/network differences from other nodes. thanks On Tue, Oct 16, 2018, 9:51 AM Karthik

Re: [EXT] Re: Cluster Warnings

e see this warning. > > > > -Karthik > > > > *From:* Joe Witt [mailto:joe.w...@gmail.com] > *Sent:* Tuesday, October 16, 2018 7:55 AM > *To:* users@nifi.apache.org > *Subject:* [EXT] Re: Cluster Warnings > > > > the logs show the fourth node is the slowe

Re: Problem with remote groups reconnecting to input port after restart.

bill can you share more about what is observed on restart? backlogs? can you share screenshots and or logs. thanks On Tue, Oct 23, 2018, 7:28 AM William Gosse wrote: > Hi, > > > > I’m trying to implement a Listfile>FetchFile pattern similar to the one > described on the following link: > >

Re: Fetch/Put via FTPS?

Katrina, Nobody has added FTPS support as of yet that I am aware of into NiFi. We have FTP or SFTP. Thanks On Wed, Oct 24, 2018 at 1:48 PM Katrina Henrie wrote: > > There's an old thread on here relating to FTPS (FTP with TLS/SSL) - > http://apache-nifi.1125220.n5.nabble.com/FTPS-td19292.html

Re: ScriptedRecordReader Error Handling

Cant your logic detect the strange characters and then apply its behavior? Alternatively, you could perhaps use ValidateRecord and have its reader only understand the good records. It should kick out the bad records and you can then apply deeper processing on them. Thanks On Fri, Oct 26, 2018

Re: nifi issue with sftp server

how many connections does your sftp server allow at once and how many threads is nifi given for communication? On Mon, Oct 29, 2018, 7:10 AM Alfredo De Luca wrote: > Hi all. We have some issue with our nifi (v. 1.6) where occasionally we > have an error saying ... > Auth fail. > > On the

Re: [EXT] Re: Cluster Warnings

m HTTP Request for NIFI03:8443 took 5 millis > > Perform HTTP Request for NIFI02:8443 took 5 millis > > Perform HTTP Request for NIFI01:8443 took 224 millis > > Phase Completed for All Nodes took 224 millis > > Notifying All Threads that Request is Complete took 0 millis >

Re: nifi issue with sftp server

h.JSchException: Auth fail: {} >>> >>> >>> On Mon, Oct 29, 2018 at 12:27 PM Alfredo De Luca >>> wrote: >>>> >>>> Hi Joe. >>>> Sftp server allows 200 connections... not sure about the nifi. I ll check >>>> >>>>

Re: ConsumeKafkaRecord won't pull new events from Kafka

Mike, In previous versions that processor didn't have a parse.failure relationship I believe. In that case data would not be pulled when unable to parse it. In newer versions of the processor it has it. So check there first. Thanks On Tue, Nov 13, 2018 at 9:56 AM Mike Thomsen wrote: > > I

Re: ConsumeKafkaRecord won't pull new events from Kafka

> (spring boot service, not NiFi) was incorrectly building the binary form of > the Avro (didn't have embedded schema). > > Mike > > On Tue, Nov 13, 2018 at 12:33 PM Joe Witt wrote: >> >> Viking >> >> Just to clarify it isn't that it is trying t

Re: ConsumeKafkaRecord won't pull new events from Kafka

Viking Just to clarify it isn't that it is trying to merge records to create larger flow files but rather it is 'playing the kafka interaction as it lies' so to speak. When polling from Kafka we can get one or more records. We're just taking advantage of that without trading off speed. Thanks

Re: [EXT] Re: Cluster Warnings

> Perform HTTP Request for NIFI04:8443 took 2 millis > > Perform HTTP Request for NIFI03:8443 took 5 millis > > Perform HTTP Request for NIFI02:8443 took 5 millis > > Perform HTTP Request for NIFI01:8443 took 224 millis > > Phase Completed for All Nodes took 224 millis > &

Re: Anonymization of CSV data with NiFi?

Hello You'd have to write custom code for the anonymization logic specifically. But it would be a perfect case for the Record oriented processors. You could have a pretty simple "AnonymizeRecord" processor for example. Thanks On Mon, Nov 12, 2018 at 5:31 PM wrote: > > Hi there, > > I have a

Re: rotating servers

Tom, Just want to make sure I'm following here but the custom Controller Service you reference is where this logic lives/should live. Right? If that is the case then you'll want to focus on improving the behavior of the controller service to switch over. If that is not the case can you share

Re: Dynamic Mapping

Ryan I am not entirely sure I fully understand the scenario so read the following with a bit of caution. But the gist I think i'm reading is that system A generates data against a given schema referenceable with some guid say 'GUID1'. For every GUID1 from System A there is a similar/but

Re: ConsumeMQTT processor causing resource issues when deployed in MiNiFi

Certainly could be that memory space on the machine is constrained enough that creation of a new thread isn't possible. Could be a resource leak. To Aldrin's point though I'd say the thread dump would be really helpful. In fact, I'd recommend grabbing a few at different time intervals such as

Re: NiFi for IoT Analytics

Julian, It is a great question. If you have not already read them the overview [1] and life of a flow file [2] docs will probably help orient things. One is really high level and the other is much lower level so they might not provide enough clarity. For your question about what happens to

Re: Nifi RecordReader for Json?

hello v there are some excellent blogs on nifi record reader and related items. you definitely want to check them out to understand how it works. https://blogs.apache.org/nifi/entry/record-oriented-data-with-nifi thanks On Thu, Aug 30, 2018, 8:48 AM l vic wrote: > So, where's

Re: FetchS3 not fetching all objects?

hello i think we would need to see directory listing from s3, state info and config from nifi, and logs. you can step through in a debug session to see what the proc sees as well. thanks On Fri, Sep 21, 2018, 6:55 PM Vets, Laurens wrote: > Is there a way to troubleshoot this further? I did

Re: Wait/Notify inconsistent behavior

Please avoid sending more copies of the question. Hopefully someone familiar with the processors in question will be available in time. Thanks On Fri, Jan 4, 2019 at 9:14 PM Luis Carmona wrote: > Hi everyone, > > Im having a strange behavior with Wait / Notify mechanism. Attached is > the

Re: Wait/Notify inconsistent behavior

gt; Thanks for letting me know. > > LC > > ---------- > *De: *"Joe Witt" > *Para: *"users" > *Enviados: *Viernes, 4 de Enero 2019 23:23:02 > *Asunto: *Re: Wait/Notify inconsistent behavior > > Please avoid sending more copies of

Re: Kafka max topics

Ah wow. It was definitely arbitrary We could make it configurable and arbitrarily larger by default :) Please file a JIRA. Good catch as this makes the thread the other day make a lot more sense. thanks On Tue, Jan 22, 2019 at 4:06 PM Boris Tyukin wrote: > Hi guys, > > does anyone know

Re: How to get Node ID of disconnected node to be used in a cluster/node monitoring flow

ocs/html/administration-guide.html#proxy_configuration > <https://urldefense.proofpoint.com/v2/url?u=https-3A__nifi.apache.org_docs_nifi-2Ddocs_html_administration-2Dguide.html-23proxy-5Fconfiguration=DwMFaQ=gJN2jf8AyP5Q6Np0yWY19w=MJ04HXP0mOz9-J4odYRNRx3ln4A_OnHTjJvmsZOEG64=Fu5GvEY5W6MSQUzs1kOAlkIcfLF

Re: NiFi suddenly releases a lot of disk space on shutdown

good catch mike. that is def a thing. historically the jvm had issues with memory mapped io as well not being able to let go of files until restart. On Wed, Dec 12, 2018, 7:19 PM Mike Thomsen Mike, > > I did lsof +L1 and saw a ton of files listed that were marked (deleted), > but the OS was

Re: NIfi cluster on Docker

Hello Not aware of folks working with AWS/Fargate but I am aware of a slew of NiFi on K8S work including NiFi on EKS...so I know that works well. Curious to hear your progress. Thanks Joe On Tue, Dec 18, 2018 at 7:57 AM Jean-Sebastien Vachon wrote: > > Hi all, > > > anyone has tried running a

Re: nifi 1.8 cluster load balancer properities

Hello Have you reviewed this doc for load balancing properties? If you are using a 1.7.1 or earlier properties file youll want to diff with 1.8 to see what is new. http://nifi.apache.org/docs/nifi-docs/html/administration-guide.html thanks On Tue, Dec 4, 2018, 10:02 AM Vishal Jadhav

Re: How to get Node ID of disconnected node to be used in a cluster/node monitoring flow

chad in 1.8.0 you should not see much need to script that. are you still seeing disconnects in 1.8.0 that it doesnt restore on its own. thanks On Mon, Dec 10, 2018, 10:08 AM Woodhead, Chad Hey Jeremy! > > > > We occasionally see a node disconnect from the cluster (not disappear from > the

Re: hbaseclient service is failed to enable

Ravi it is very tough to read that stack trace as provided but the core piece of it seems to be that authorization is failing for "/hbase". Have you confirmed that the proper authorization configuration is present? Thanks On Sun, Nov 25, 2018 at 4:50 PM Ravi Papisetti (rpapiset) wrote: > > Hi,

Re: NiFI as Data Pipeline Orchestration Tool?

Jon First things first - Sonos is awesome. Now back to the matter at hand... NiFi is quite often used for various forms of orchestration of other systems doing their thing. However, I'll state that isn't really its primary purpose so for pure orchestration cases it can leave you with a less

Re: Run another PG once first one is finished

Boris, It is of course a little bit interesting to state that two groups are completely independent while then also stating you want to trigger group B on the completion of group A. Clearly they have a dependency. I dont quite follow the desire then to decouple them from the flow configuration

Re: Mergecontent on Cluster

Hello NiFi lets you merge data based on some correlation value. So your case you seem to want that correlation to be the combined key of 'type + + mm'. This is fine/easily configured. NiFi lets you merge data based on reaching the min/max size requirements or based on time based kick out.

Re: NiFi 1.9.1 release contains a bug causing content repos to fill...

through this week. Thanks On Tue, Apr 2, 2019 at 12:17 PM William Gosse wrote: > Also can someone explain more what this issue is and how serious it is. > > Can a patch be made available to fix this issue for existing 1.9.1 > installations? > > > > *From:* Joe Witt >

NiFi 1.9.1 release contains a bug causing content repos to fill...

All, We will work to promptly produce an Apache NiFi 1.9.2 release which will correct a regression introduced in the 1.9.1 release. The issue, NIFI-6150 has been identified and fixed on master and once further testing confirms we're good we'll get the RC going. I've updated the release guide to

Re: Weird ListFile Issue

William What was the real dir name vs the erroneous name? What might happens depends on many factors such as os behaviors. thanks On Fri, Mar 22, 2019, 12:47 PM William Gosse wrote: > I ran into kind of a weird issue with the ListFile processor. I was > referencing a variable for my input

Re: PutKafka use with large quantity of data?

Hello There isn't really a feedback mechanism based on load on the Kafka topic. When you say overrunning the topic do you mean that you don't want there to be a large lag between consumers and their current offset and if that grows you want NiFi to slow down? I dont believe there is anything

Re: PutKafka use with large quantity of data?

; rate contol) > > > On Thu, Apr 4, 2019 at 10:42 AM Joe Witt wrote: > >> Hello >> >> There isn't really a feedback mechanism based on load on the Kafka >> topic. When you say overrunning the topic do you mean that you don't want >> there to be a larg

Re: insufficient content written

Hello Can you share logs or screenshots or anymore details to illustrate what you're seeing? Thanks On Fri, Apr 5, 2019 at 2:14 PM Jean-Sebastien Vachon wrote: > Hi all, > > I've added an UpdateAttribute processor to change the filename attribute > and since then I am not able to either view

Re: NiFi 1.9.1 release contains a bug causing content repos to fill...

.2 RC? > > > > Cheers, > > Mathieu. > > > > *From:* Joe Witt [mailto:joe.w...@gmail.com] > *Sent:* Wednesday, March 27, 2019 11:00 AM > *To:* users@nifi.apache.org > *Subject:* NiFi 1.9.1 release contains a bug causing content repos to > fill... > > &

Re: When do I download the older releases of Apache NiFi

Vijay, The downloads page is meant to link to the current and previous release and otherwise point to the archives. This is generally something the ASF requests because the current links are served by mirrors whereas older ones come from ASF infra/archives and have stricter bandwidth/rate

[ANNOUNCE] Apache NiFi 1.9.0 release

Hello The Apache NiFi team would like to announce the release of Apache NiFi 1.9.0. In addition to more than 100 improvements and bug fixes, this release makes integration with Apache Kudu and Impala a breeze. Provides stronger integration with Google Big Query and Amazon Web Services. New

Re: Proposal for ElasticSearch support

...probably for dev thread but I am starting to think we should just start removing certain nars from the convenience build/assembly and documenting it in the migration guide for users that need those. We can then show them how to use the hot loading/etc.. Thanks On Wed, Feb 20, 2019 at 10:04

Re: merging flowfiles?

Hello You could put a funnel for your two jsonpath processors to send to then the funnel to mergecontent. That at least addresses the multi input paths comment. Whether your data can simply be merged like this or not is possibly another matter but I presume you have that in hand. Thanks On

Re: Getting exception with multiple DBCPConnectionPools: sqljdbc_auth.dll already loaded in another classloader

Brad What you’re running into is that the same native lib, by name, cannot be loaded more than once in the entire jvm. Since the component you’re using is loading the missal driver in two diff processors we are hitting this. We can work around this limitation of jvm classloader isolation by

Re: Nifi 1.8.0 Very Slow NAR Unpacking

John it is too hard to guess from just the shared images. A full stack trace and set of logs may be more enlightening. If unzipping/opening such a thing is so incredibly slow there are two culprits I've heard of/seen in the past. 1) Anti-virus/security related software that is overly

Re: Different NiFi Node sizes within same cluster

This is generally workable (heterogenous node capabilities) in NiFi clustering. But you do want to leverage back-pressure and load balanced connections so that faster nodes will have an opportunity to take on the workload for slower nodes. Thanks On Wed, Mar 6, 2019 at 3:48 PM James Srinivasan

Re: join two datasets

Right I agree with Bryan so let me expand a bit. There are some key primitives that stream processing systems address as it relates to joining two live streams that those systems are designed to solve well. NiFi offers nothing special/unique in that space. Now, as Bryan pointed out a really

Re: join two datasets

I should add you can use NiFi to update the reference dataset in a database/backing store in one flow. And have another flow that handles the live stream/lookup,etc. MarkPayne/Others: I think there are blogs that describe this pattern. Anyone have links? On Fri, Feb 22, 2019 at 12:27 PM Joe

Re: join two datasets

Boris, Great. So have a process to load the periodic dataset into a lookup service. COuld be backed by a simple file, a database, Hive, whatever. Then have the live flow run against that. This reminds me - we should make a Kudu based lookup service i think. I'll chat with some of our new Kudu

Re: Export/Import flow files

Eric, You could download the flowfile content using the provenance click-to-content feature. You can see the attributes there too if you need to recreate those. You can then ingest that file using GetFile or something on the other system. There are additional built-in mechanisms that allow you

Re: Registry 1.8.0 flows fail to import into 1.9.1 node

glad you found how to get through it - we should make the user experience better. What led you to find the solution? On Thu, Mar 14, 2019 at 4:33 PM Mikhail Rolshud (BLOOMBERG/ 120 PARK) < mrols...@bloomberg.net> wrote: > Ignore this - it had nothing to do with registry or versions. > > Looks

Re: Most efficient means to search for a character in flowFiles

James For the problem as you described it the processor you definitely want is https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.0/org.apache.nifi.processors.standard.ScanContent/index.html It is a rather impressively fast implementation of a string search

<    1   2   3   4   5   6   7   8   9   10   >