[
https://issues.apache.org/jira/browse/NIFI-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18015226#comment-18015226
]
Joe Witt commented on NIFI-14882:
---------------------------------
Record oriented processing does add overhead no doubt.
But there are important guarantees of the data/handling and framing we do have
to honor in many cases.
In cases you can simply truly bypass record handling then you can use the
demarcator strategy. But we're not looking to add more data types there in
ConsumeKafka directly.
You note data rates which would be 1.2 MB/sec when using some record based
config. And 58MB/sec when using non-records or what this PR brings. To be
clear neither rate but especially the 1.2MB/sec is particularly strong. We
routinely, even with records, are well beyond those rates.
So I recommend we first go deeper into what configuration were you using to
achieve 1.2MB/sec.
The PR as-is is not something we would want to advance. Lets focus more on the
problem and then unpack what next steps might make sense.
Thanks
> ConsumeKafka improvement for Avro+SchemaRegistry input
> ------------------------------------------------------
>
> Key: NIFI-14882
> URL: https://issues.apache.org/jira/browse/NIFI-14882
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
> Affects Versions: 2.5.0
> Environment: NiFi 2.4+, Kafka 3
> Reporter: Alex Goos
> Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> When Kafka receives Avro records marked with a schema identifier in the
> Confluent SchemaRegistry - and no immediate transformation is needed - the
> costly conversion into NiFiRecords and back to Avro is not needed. The
> performance can be substantially improved by simply batching records from the
> same source and with the same schema into a Avro Datafile.
> A new ProcessingStrategy "Avro Datafile" can be added to the existing ones.
> In our setup, a VM with a Intel(R) Xeon(R) CPU E5-2695 v4 @ 2.10GHz CPU, and
> a single executor thread for ConsumeKafka this bumps throughput from
> 500MB/5Min (Precessing Strategy RECORD) to 17GB/5min
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
