These db rows are fairly small. Five or six small text fields and five or six integer fields plus a timestamp. Looking at the queue in NiFi, about 440 bytes each.
If I exported the template correctly, the flow should show that I am trying to generate a set of query statement flow files, each 10k records, and then executesql on them. Next I use SplitAvro and AvroToJSON to get a set of flow files, each one record in json format. Those are what I'm feeding in to Kafka. -Daniel On Oct 30, 2016 11:00 AM, "Joe Witt" <[email protected]> wrote: Daniel How large is each object you are trying to write to Kafka? Since it was working based on a different source of data but now problematic that is the direction I am looking in in terms of changes. Output of the db stuff could need demarcation for example.
