[GitHub] nifi issue #2737: NIFI-5231 Added RecordStats processor.

2018-06-10 Thread ijokarumawak
Github user ijokarumawak commented on the issue: https://github.com/apache/nifi/pull/2737 For future work, I've submitted this to add stats for numerical values. https://issues.apache.org/jira/browse/NIFI-5291 ---

[GitHub] nifi issue #2737: NIFI-5231 Added RecordStats processor.

2018-06-10 Thread ijokarumawak
Github user ijokarumawak commented on the issue: https://github.com/apache/nifi/pull/2737 @MikeThomsen Thanks for the updates. LGTM, +1! Merging. ---

[GitHub] nifi issue #2737: NIFI-5231 Added RecordStats processor.

2018-06-07 Thread MikeThomsen
Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2737 @ijokarumawak Anything to add now or can we close this out? ---

[GitHub] nifi issue #2737: NIFI-5231 Added RecordStats processor.

2018-06-04 Thread ijokarumawak
Github user ijokarumawak commented on the issue: https://github.com/apache/nifi/pull/2737 @MikeThomsen Thanks for the updates. I will continue more close review on this when I have time, probably tomorrow. ---

[GitHub] nifi issue #2737: NIFI-5231 Added RecordStats processor.

2018-06-02 Thread MikeThomsen
Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2737 @ijokarumawak can you review? ---

[GitHub] nifi issue #2737: NIFI-5231 Added RecordStats processor.

2018-05-24 Thread MikeThomsen
Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2737 I'm not too familiar with the deep internals of the framework either. What we've seen is that with the records API it just makes sense to leverage the provenance system because it already tracks t

[GitHub] nifi issue #2737: NIFI-5231 Added RecordStats processor.

2018-05-24 Thread ottobackwards
Github user ottobackwards commented on the issue: https://github.com/apache/nifi/pull/2737 That makes sense, just thinking it through, and obviously I don't understand everything as well ;) I guess I never thought of provenance as including perf and stats stuff, so it seems like p

[GitHub] nifi issue #2737: NIFI-5231 Added RecordStats processor.

2018-05-24 Thread MikeThomsen
Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2737 Not sure if I like that approach. The beauty of what we did was it just puts the data into the provenance repository and there aren't that many flowfiles to track. Maybe a few hundred thousand ove

[GitHub] nifi issue #2737: NIFI-5231 Added RecordStats processor.

2018-05-24 Thread ottobackwards
Github user ottobackwards commented on the issue: https://github.com/apache/nifi/pull/2737 prob the reader and write would have to get some context passed where they can track states or increment stats, then be configured with a 'reporting' task to send the stats from a given context

[GitHub] nifi issue #2737: NIFI-5231 Added RecordStats processor.

2018-05-24 Thread MikeThomsen
Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2737 I'm not sure how that'd work out because you need to actually read the flowfiles and calculate the stats. ---

[GitHub] nifi issue #2737: NIFI-5231 Added RecordStats processor.

2018-05-24 Thread ottobackwards
Github user ottobackwards commented on the issue: https://github.com/apache/nifi/pull/2737 @MikeThomsen this is cool. The only thing it makes me wonder is if this kind of data can't be automatically generated and sent to a repository, almost like a new ( or actual ) reporting