Wow. Really nice work. Thank you so much Adam. I can honestly say I couldn't have done it better myself.
Kyle -------- Original message -------- From: Adam Lamar <adamond...@gmail.com> Date:01/31/2016 1:57 AM (GMT-05:00) To: users@nifi.apache.org Cc: Subject: Re: ListS3 processor? Kyle/Joe, I've been meaning to document this process myself, and just finished a post with some details: https://adamlamar.github.io/2016-01-30-monitoring-an-s3-bucket-in-apache-nifi/ Hope that helps, Adam On 1/30/16 9:29 PM, Joe Witt wrote: > Kyle, > > The ideal case for communicating how to do this would be both a > template and an associated doc. Great for a blog or wiki page or > something. We can of course give you perms to write to a wiki page on > the nifi wiki if interested. The template itself can also be > annotated with comments that show up right in the flow itself. That > may be a fine option too. > > Thanks > Joe > > On Sat, Jan 30, 2016 at 2:52 PM, Kyle Burke <kyle.bu...@ignitionone.com> > wrote: >> Joe/Joe, >> Thanks for the response. It makes sense to use SNS and SQS to respond to >> S3 file changes. I’m going see if my company will give me access to those >> Amazon services. I found an article that explains how to setup on this >> functionality in the Amazon console. Once that’s setup it seems pretty >> straight forward to use GetSQS/DeleteSQS. I suspect many will want this >> functionality but I’m not sure what’s the best method (i.e. Template or user >> doc) that explains how to solve this in nifi. I’ll be happy to submit >> something if you let me know the right method. >> >> http://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html >> >> Respectfully, >> >> Kyle Burke | Data Science Engineer >> IgnitionOne - Marketing Technology. Simplified. >> Office: 1545 Peachtree St NE, Suite 500 | Atlanta, GA | 30309 >> >> >> From: Joe Witt >> Reply-To: "users@nifi.apache.org" >> Date: Saturday, January 30, 2016 at 2:06 PM >> To: "users@nifi.apache.org" >> Subject: Re: ListS3 processor? >> >> Kyle >> >> Let us know if that doesn't get you what you need. We have a decent set of >> templates but I didn't see one that demonstrates interaction with amazon >> services. >> >> Thanks >> Joe >> >> On Jan 30, 2016 12:56 PM, "Joey Frazee" <joey.fra...@icloud.com> wrote: >>> Kyle, >>> >>> I think you can do what you want right now without ListS3 by using S3 >>> event notifications. You can configure an event notification to publish to >>> SQS and then use GetSQS to retrieve the events and FetchS3Object to get the >>> JSON file and the rest of the flow could be written as you have in mind. >>> >>> Depending on your scale, this might be preferable because it's >>> slow/expensive to do listings on S3 prefixes that have a lot of file >>> matches. >>> >>> >>> -joey >>> >>> On Jan 30, 2016, at 11:40 AM, Joe Skora <jsk...@gmail.com> wrote: >>> >>> Kyle, >>> >>> Processors exist to Put, Fetch, and Delete S3Objects, but ListS3 is in the >>> backlog on ticket NIFI-840 at the moment. It should fit the List/Fetch >>> metaphor like the List/Fetch processors pairs for xFile, xHDFS, xSFTP, etc. >>> >>> Regards, >>> Joe Skora >>> >>> On Sat, Jan 30, 2016 at 10:14 AM, Kyle Burke <kyle.bu...@ignitionone.com> >>> wrote: >>>> All, >>>> I'm trying to get Nifi set up to a move data around S3. My first >>>> attempt is to just monitor a S3 folder where json files are placed and then >>>> copy the file, convert it to Avro, and the drop it in a different S3 >>>> folder. >>>> The documentation is pretty slim for working with S3. I can't seem to get >>>> it >>>> working and was wondering if anyone had any S3 examples for monitoring an >>>> S3 >>>> folder (i.e.. something like a ListS3 processer similar to what is >>>> available >>>> on a local file system?) >>>> >>>> Respectfully, >>>> >>>> Kyle Burke | Data Science Engineer >>>> IgnitionOne - Marketing Technology. Simplified. >>>> Office: 1545 Peachtree St NE, Suite 500 | Atlanta, GA | 30309 >>>> Direct: 404.961.3918 >>>>
