Issue with GenerateTableFetch bulk load when using MS SQL Server database type

2017-07-24 Thread Bryan Quinn
Hi, I'm trying to use the GenerateTableFetch component for 'MS SQL 2012+' database - for a full table export each time its run. However I'm forced to set a column name in the 'Maximum-value Columns' field when the component is run (error only shows during execution if I omit this setting). I'm s

Re: Issue with GenerateTableFetch bulk load when using MS SQL Server database type

2017-07-24 Thread Matt Burgess
Bryan, This is likely a bug, I will investigate and write it up if so. In the meantime, are you planning on doing the bulk fetch in parallel (with a Remote Process Group into ExecuteSQL across a NiFi cluster)? If not, you may find that QueryDatabaseTable is a good alternative, it can be configu

RE: Issue with GenerateTableFetch bulk load when using MS SQL Server database type

2017-07-24 Thread Bryan Quinn
Hi Matt. I’ll have a look at option a. What’s the timeframe for a fix and prod release for this assuming it’s a bug? Ballpark. Thanks, Bryan From: Matt Burgess [mailto:mattyb...@gmail.com] Sent: Monday 24 July 2017 14:40 To: users@nifi.apache.org Subject: Re: Issue with GenerateTableFetch bulk l

Re: ListS3 duration

2017-07-24 Thread Laurens Vets
Hi Jeff, Did you by any chance found your flow? I want to compare yours against mine if possible. Basically, what I came up with is: 1. ListS3. 2. RouteOnAttribute. Route only when the filename contains "_CloudTrail_". I don't care about the digests (I think). 3. FetchS3Object. 4. Updat

Re: ListS3 duration

2017-07-24 Thread Joe Skora
Lauren, For debugging and tracking purposes, you might want to look at the fragment attributes created by SplitJson instead of generating a UUID to add to each split. The "fragment.identifier" contains a unique UUID shared among all fragments, "fragment.index" indicates the index of current split

NiFi 1.3: Simplest way possible of creating CSV files from SQL queries

2017-07-24 Thread Márcio Faria
Hi, I'm looking for the simplest way possible of creating CSV files from SQL queries using Apache NiFi 1.3.  The flow I currently have (the files are to be SFTP'ed to a remote server): ExecuteSQL -> UpdateAttribute -> ConversionRecord [3 CSs] -> PutSFTP The concept of SchemaRegistry is new to me