jozahner commented on code in PR #10829:
URL: https://github.com/apache/nifi/pull/10829#discussion_r2743699455
##########
nifi-extension-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/avro/AvroReader.java:
##########
@@ -60,12 +60,27 @@ public class AvroReader extends SchemaRegistryService
implements RecordReaderFac
.required(true)
.build();
+ static final PropertyDescriptor FAST_READER_ENABLED = new
PropertyDescriptor.Builder()
+ .name("Fast Reader Enabled")
+ .description("""
+ When enabled, the Avro library uses an optimized reader
implementation that improves read performance
+ by creating a detailed execution plan at initialization.
However, this optimization can lead to
+ significantly higher memory consumption, especially when
using schema inference. If OutOfMemory errors
+ occur during Avro processing, consider disabling this
option.""")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .allowableValues("true", "false")
+ .defaultValue("true")
Review Comment:
To be honest, I don't know what the OOM message was in detail - we don't
have the logs anymore. I've asked Daniel Beicht in the Slack Channe, he faced
most likely the same issue.
We are reading the prometheus metrics 'nifi_jvm_heap_usage' and
'nifi_jvm_heap_used', so I would assume the heap was the issue.
We have two clusters which are doing mostly the same, one has only inferred
schemas and one has a mix of inferred schemas (QueryRecord) and a
ListenUDPRecord processor with a Record Writer which has a static schema. Both
are using only a handful processors which are dealing with AVRO flowfiles.
Regarding performance, we just changed the fastread behavior. We will
monitor the CPU usage the next few days and I can report back whether I see a
difference between NiFi 2.5.0 and 2.7.2 (but with fastread disabled)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
