Endre Kovacs created NIFI-6180: ---------------------------------- Summary: DruidTranquilityController: expose `firehoseGracePeriod` for higher index task throughput Key: NIFI-6180 URL: https://issues.apache.org/jira/browse/NIFI-6180 Project: Apache NiFi Issue Type: Improvement Components: Extensions Affects Versions: 1.9.1 Reporter: Endre Kovacs Assignee: Endre Kovacs
During integration testing NiFi with Druid, I noticed a constant, overhead in the realtime index tasks: eg.: even if i set the {code:java} druid-cs-window-period=PT1M{code} the task duration was still at 426726ms (~7 minutes) 5 of that 7 minutes were due to firehose's grace period. My suggestion is to expose `druidBeam.firehoseGracePeriod` ([https://github.com/druid-io/tranquility/blob/master/docs/configuration.md#properties]) to the controller service: DruidTranquilityController, and build it into [https://github.com/apache/nifi/blob/3696b5bfcf0bd9e12ee4e9472f3413a93c9c0fcd/nifi-nar-bundles/nifi-druid-bundle/nifi-druid-controller-service/src/main/java/org/apache/nifi/controller/druid/DruidTranquilityController.java#L452-L457] the beam config. Currently this constant takes the default value of 5 minutes, causing a constant overhead on each of the indexing tasks initiated by NiFi, no matter how few/small the workload window. -- This message was sent by Atlassian JIRA (v7.6.3#76005)