Hi Sahil, Disclaimer: I am not using that external component ;-) It seems, that it is possible to pass extra consumer properties via "consumerPropertiesFileName", which uses a util relying on a plain file input stream to load a properties file. If that doesn't work, it sounds like a bug (or the location of the properties file is an issue).
Gruß Richard Am Samstag, dem 30.09.2023 um 12:41 +0530 schrieb Sahil Kamboj: > Hi Richard, > > First of all Thanks for responding and sorry for not putting more > context.Below are the answers to your question. > > We are using strom 2.4.0. By lag I mean the difference between spout > committed offset value and log end offset value. We are using the > KafkaSpout component of the storm ecosystem. Strom does report lag on > storm ui when we visit the summary page of a topology, where it > reports lag per partition. > > ISSUE - > It works well with non acl enabled kafka cluster. But with ACL > enabled kafka cluster we get exception on storm ui. > Unable to get offset lag (Kafka Consumer exception.). > After following apache storm doc we have placed storm_jaas.conf under > the ${STORM_HOME_DIR}/conf directory. But still lag was not reported > on ui as ui got stuck. It never loads on the topology summary page. > On navigating same master node and doing ps -aux | grep java we can > see the kafka lag monitor process started but it never ends. We have > to kill the process forcefully using kill -9. > Process is passed with following args > -t <topic name> > -g <group name> > -b <brokers> > -s <security protocol> = SASL_PLAINTEXT in our case. > > The Fix - > For to make it work we have builded a custom jar of kafka-monitor. > Below is the code fix that we have applied > class - KafkaOffsetLogUtil.java > method - getOffsetLags > > if (newKafkaSpoutOffsetQuery.getSecurityProtocol() != null) { > props.put("security.protocol", > newKafkaSpoutOffsetQuery.getSecurityProtocol()); > props.put("sasl.mechanism", "PLAIN"); // this could > be different as per kafka acl policy. For our use case we fixed it to > our implementation. > } > Post this lag is reported successfully on UI. SASL_MECHANISM property > set is missing in the storm kafka monitor code base. > > Question - > Now question here is that this is a standard problem and everyone > must be facing it with an ACL enabled kafka cluster. So does storm > have a standard fix for it? I navigated through > TopologySpoutLag.class code and saw it can pass extra consumer > properties through a temporary file creation but it doesn't work. > Please let me know the solution around this. > > P.S. - I have not tried storm 2.5.0 yet but the code part does not > have any change for this. I checked the master branch code and it is > the same. > > Hope it answers all your questions. Please let me know if further > info is required from me. Waiting for your response.... > > Regards > Sahil > > > > > On Fri, Sep 29, 2023 at 7:34 PM Richard Zowalla <r...@apache.org> > wrote: > > Hi, > > > > think there is more context information required? > > > > - What exactly does "lagging" means? > > - What is "the fix" in your custom JAR? > > - Which modules are you using exactly? > > - Does it also happen with 2.5.0 ? > > > > Gruß > > Richard > > > > Am Freitag, dem 29.09.2023 um 08:55 +0530 schrieb Sahil Kamboj: > > > Hi Team, > > > > > > Could you please help us with strom ui lag with acl enabled > > > kafka. > > > Currently we have to build a custom kafka monitor jar with a fix > > > to > > > get it worked properly. We are on version 2.4.0 and i can see in > > > the > > > code we do have a code snippet related to passing extra consumer > > > configs but that is not working. Could you please let us know the > > > correct way of doing this? > > > > > > PS: We have to set sasl.mechamism to the desired value in the > > > props > > > of consumer to get it worked under KafkaOffsetLagUtil.java > > > > > > Regards > > > Sahil > > > > > > On Thu, 28 Sep, 2023, 5:46 pm Sahil Kamboj, > > > <sahilkamboj...@gmail.com> wrote: > > > > > > > > > > > > On Thu, 28 Sep, 2023, 5:42 pm Sahil Kamboj, > > > > <sahilkamboj...@gmail.com> wrote: > > > > > Hi Team, > > > > > > > > > > Could you please help us with strom ui lag with acl enabled > > > > > kafka. Currently we have to build a custom kafka monitor jar > > > > > with > > > > > a fix to get it worked properly. We are on version 2.4.0 and > > > > > i > > > > > can see in the code we do have a code snippet related to > > > > > passing > > > > > extra consumer configs but that is not working. Could you > > > > > please > > > > > let us know the correct way of doing this? > > > > > > > > > > PS: We have to set sasl.mechamism to the desired value in the > > > > > props of consumer to get it worked under > > > > > KafkaOffsetLagUtil.java > > > > > > > > > > Regards > > > > > Sahil > >