[GitHub] nifi issue #2813: NIFI-5084 Added GenerateRecord processor.
Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2813 NP. ---
[GitHub] nifi issue #2813: NIFI-5084 Added GenerateRecord processor.
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/2813 Ah yup, totally different lol. I was thinking GenerateFlowFile but with just a writer + dynamic properties to set fields. Sorry for the confusion! ---
[GitHub] nifi issue #2813: NIFI-5084 Added GenerateRecord processor.
Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2813 I don't think it's compatible with your use case. What I built, based on how I read the ticket, was a processor that uses a Confluent library to generate random data based on an Avro schema. So maybe I'm missing something, but I think that's different from what you need. ---
[GitHub] nifi issue #2813: NIFI-5084 Added GenerateRecord processor.
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/2813 Is there a reason this processor wasn't just a part of the normal record bundle instead of being thought of as part of a larger reprocessing bundle? If it's just another file in that same nar, it wouldn't be a lot of space. For reference, I have the use-case where I have attributes on a FF and the content is a binary file. After putting the file to disk, I want to create a nested json object to continue processing. I essentially want to create a new record just from EL (same as UpdateRecord but with no reader). I believe GenerateRecord would fill this need? ---
[GitHub] nifi issue #2813: NIFI-5084 Added GenerateRecord processor.
Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2813 Also, the space issue is becoming a serious concern for our binary distributions. So now things that aren't really important for core have to be weighed against that consideration if you want normal users to have an easy time getting to them. ---
[GitHub] nifi issue #2813: NIFI-5084 Added GenerateRecord processor.
Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2813 @JPercivall Here you go: http://apache-nifi-developer-list.39713.n7.nabble.com/Should-I-withdraw-this-PR-td19475.html I'm fine with reopening this if you want to take over the code review. Folks have been pretty busy and it was starting to get to stale, so I pulled the trigger to include it in a larger bundle of testing tools for users. ---
[GitHub] nifi issue #2813: NIFI-5084 Added GenerateRecord processor.
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/2813 @MikeThomsen I'm looking for the discussion on the mailing list on why this was decided to not be merged in and I can't find it. I only see the "How would we handle this?" thread[1]. Could you link to it? [1] http://apache-nifi-developer-list.39713.n7.nabble.com/How-would-we-handle-this-td19151.html ---
[GitHub] nifi issue #2813: NIFI-5084 Added GenerateRecord processor.
Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2813 Per the discussion on the mailing list, I am moving the code [here](https://github.com/MikeThomsen/nifi-datageneration-bundle) and closing this PR. ---
[GitHub] nifi issue #2813: NIFI-5084 Added GenerateRecord processor.
Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2813 @ottobackwards 1. I wanted to keep it simple. You can't really wire up the writer without going through most of that, and I wanted to have a simple configuration option for just dumping a schema in for less experienced users who just want to get stuff done. 2. I think that would make a good follow on ticket. Some thought would be need to determine how to control the size of a record set. I could see that causing issues for processors. ---
[GitHub] nifi issue #2813: NIFI-5084 Added GenerateRecord processor.
Github user ottobackwards commented on the issue: https://github.com/apache/nifi/pull/2813 @MikeThomsen this is really great. A couple of questions: - Why wouldn't you want to have the same schema selection capabilities for both the processor (virtual reader) and the writer? - Did you consider creating a Generating Reader Service that could be re-used by this processor and possibly many others at the same time? ---
[GitHub] nifi issue #2813: NIFI-5084 Added GenerateRecord processor.
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/2813 Not sure. But we should just organize a proper effort to get an extension registry in play. It will take some time but with us not being able to make the download larger we should be properly motivated to sort that out. ---
[GitHub] nifi issue #2813: NIFI-5084 Added GenerateRecord processor.
Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2813 @joewitt so here's a thought... do you think it would be feasible to write a deployment process that pushes the extra nars (ex atlas, hive3 and this) to our GitHub repo's "releases" list? Or are there restrictions on space, etc.? ---
[GitHub] nifi issue #2813: NIFI-5084 Added GenerateRecord processor.
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/2813 Mike - i'll look when able. In the mean time anything present in the resulting Nar(s) needs to be accounted for in the L of that nar. The nar should probably not be added to the assembly for now since we're space constrained and it is for test purposes and someone could easily add it when needed. Therefore, no need to update the nifi-assembly/L thanks ---
[GitHub] nifi issue #2813: NIFI-5084 Added GenerateRecord processor.
Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/2813 Sure @MikeThomsen I'll take a look ---
[GitHub] nifi issue #2813: NIFI-5084 Added GenerateRecord processor.
Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2813 As mentioned in the docs, the Confluent lib that I use for this uses extensions to Avro syntax to define the rules. ---
[GitHub] nifi issue #2813: NIFI-5084 Added GenerateRecord processor.
Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2813 @zenfenan can you review? ---
[GitHub] nifi issue #2813: NIFI-5084 Added GenerateRecord processor.
Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2813 @joewitt can you review the L? I added the new stuff, but wasn't sure if the existing dependencies that are covered elsewhere (Ex. Jackson and Bouncy Castle) had to also get added to the L here. ---
