Re: Processing multiple lines per flowfile with ExtractGrok

Thanks Joe and Bryan. The setup is little more involved but I was able to get ConvertRecord running with a grok reader and a json writer. And I can confirm that setup splits records as expected by newline. Nice touch to have multiple records contained in the same flow file! Thanks for the tip and

Processing multiple lines per flowfile with ExtractGrok

Hi there, I've been playing with the ExtractGrok processor and noticed I was missing some data that I expected to be extracted. After some investigation, it seems that ExtractGrok extracts only the first line of the flowfile content, and ignores the rest. Is this expected behavior? I should be

Re: NiFi ram usage

or MongoDB it's entirely possible that you >>> actually don't have enough memory to give even 512MB to NiFi. >>> >>> My recommendation would be 4GB of RAM for the VM with Xms1G and Xmx2G >>> for the heap sizes. That's very reasonable for experim

Re: NiFi ram usage

Jeff, This was a new installation so I actually hadn't set up any flows yet. NiFi wouldn't start immediately after installation (before I could configure any flows) because the system had too little ram. The 1.1GB figure is private (RSS) memory usage, which exceeded the 1GB instance limit (and

NiFi ram usage

Hi everybody, I recently started up a new cloud Linux instance with 1GB of ram to do some quick tasks in NiFi. I noticed NiFi kept dying without much information in the logs - it just seemed to stop during startup. Eventually I realized the system was running out of memory and OOM killing the

Re: ListS3 question

Laurens, Just to add slightly to this question: > Will ListS3 keep state correctly here for all 3 subdirectories? The answer is yes - ListS3 will keep state correctly for all 3 subdirectories. For example, if you setup a new ListS3 processor, give it a bucket and prefix, and start the

Re: ListS3 duration

No problem at all, glad you got it sorted out! The delimiter functions more like a filter, where all subkeys below the prefix with the delimiter are ignored. A good explanation is found here: http://docs.aws.amazon.com/AmazonS3/latest/dev/ListingKeysHierarchy.html And a quote from that link:

Re: ListS3 duration

Laurens, Sounds like you had some success using the prefix "AWSLogs//CloudTrail/ap-northeast-1/2017/07/03/", is that right? If so, when you right-click and choose "View State", is the timestamp and latest key name stored? Try changing the prefix to "AWSLogs//CloudTrail/ap-northeast-1/2017/07/",

Re: ListS3 duration

Hi Laurens, What NiFi version are you running? There was an issue where ListS3 would spin like that on buckets with many files, but it was fixed in version 1.1.0 IIRC. Hope that helps, Adam On Thu, Jul 20, 2017 at 10:05 AM, Laurens Vets wrote: > Hello, > > I'm trying to

Re: How to reject S3 Writes if folder does not exist?

Jim, Also keep in mind that as an object store, S3 uses "directories" only as a grouping concept, and not as a hierarchal storage mechanism. That's why the initial PutS3Object doesn't fail with a new "directory". See http://docs.aws.amazon.com/AmazonS3/latest/UG/FolderOperations.html I think

Re: Expression language and UTC millis

Thanks all for your help! Once I built the latest master, specifying 'UTC' as the second argument of toDate() did the trick: ${timestamp:toDate("-MM-dd'T'HH:mm:ss'Z'", "UTC"):toNumber()} Much appreciated, Adam

Re: Expression language and UTC millis

Oleg, A unix epoch timestamp is explicitly defined as the number of seconds (or millis) since Jan 1 1970 *UTC*, not any local timezone. Here's an example ruby expression from both systems correctly returning the 1487804483000 value despite their timezone settings: irb(main):003:0>

Expression language and UTC millis

Hi, I recently noticed some issues with time parsing in the expression language. On my Linux server configured with UTC time, using UpdateAttribute to convert a timestamp value to millis works as expected. Attribute name: timestamp Sample value: 2017-02-22T23:01:23Z UpdateAttribute is

Re: ListS3 processor?

I agree. There are also one-time actions like processing data inside an existing bucket for which a ListS3 processor would be well suited. Thanks for all the encouraging feedback on the blog post! Adam On 2/1/16 9:39 AM, Tony Kurc wrote: Joe, There is the possibility of people using S3

Re: ListS3 processor?

Kyle/Joe, I've been meaning to document this process myself, and just finished a post with some details: https://adamlamar.github.io/2016-01-30-monitoring-an-s3-bucket-in-apache-nifi/ Hope that helps, Adam On 1/30/16 9:29 PM, Joe Witt wrote: Kyle, The ideal case for communicating how to do

Re: "Processor requires an upstream connection" for FetchS3Object?

template around, but I'd be willing to share my flow if that would be helpful. Cheers, Adam On 1/12/16 8:14 PM, Russell Whitaker wrote: On Tue, Jan 12, 2016 at 7:12 PM, Adam Lamar <adamond...@gmail.com> wrote: On 1/12/16 8:04 PM, Bryan Bende wrote: In the case of FetchS3, there is defi

Re: GetSQS causes high CPU usage

On 11/3/15 2:41 AM, Joe Witt wrote: Adam, Just wanted to follow up on this. Have you had any better results and should we put a JIRA in behind what you're seeing? Thanks Joe On Tue, Oct 20, 2015 at 7:58 PM, Adam Lamar <adamond...@gmail.com> wrote: Adam, Thanks for the reply! Amazon su

Re: GetSQS causes high CPU usage

Amazon to check for data. It's not exactly like your standard message broker in that you have to force the poll. Anyway, throw a wait time in there and see if that fixes it. This will also help lower your monthly Amazon bill... Adam On Oct 19, 2015, at 11:41 PM, Adam Lamar <adam