Hi Sahil,
Disclaimer: I am not using that external component ;-)
It seems, that it is possible to pass extra consumer properties via
"consumerPropertiesFileName", which uses a util relying on a plain file
input stream to load a properties file. If that doesn't work, it sounds
like a bug (or the location of the properties file is an issue).
Gruß
Richard
Am Samstag, dem 30.09.2023 um 12:41 +0530 schrieb Sahil Kamboj:
> Hi Richard,
>
> First of all Thanks for responding and sorry for not putting more
> context.Below are the answers to your question.
>
> We are using strom 2.4.0. By lag I mean the difference between spout
> committed offset value and log end offset value. We are using the
> KafkaSpout component of the storm ecosystem. Strom does report lag on
> storm ui when we visit the summary page of a topology, where it
> reports lag per partition.
>
> ISSUE -
> It works well with non acl enabled kafka cluster. But with ACL
> enabled kafka cluster we get exception on storm ui.
> Unable to get offset lag (Kafka Consumer exception.).
> After following apache storm doc we have placed storm_jaas.conf under
> the ${STORM_HOME_DIR}/conf directory. But still lag was not reported
> on ui as ui got stuck. It never loads on the topology summary page.
> On navigating same master node and doing ps -aux | grep java we can
> see the kafka lag monitor process started but it never ends. We have
> to kill the process forcefully using kill -9.
> Process is passed with following args
> -t <topic name>
> -g <group name>
> -b <brokers>
> -s <security protocol> = SASL_PLAINTEXT in our case.
>
> The Fix -
> For to make it work we have builded a custom jar of kafka-monitor.
> Below is the code fix that we have applied
> class - KafkaOffsetLogUtil.java
> method - getOffsetLags
>
> if (newKafkaSpoutOffsetQuery.getSecurityProtocol() != null) {
> props.put("security.protocol",
> newKafkaSpoutOffsetQuery.getSecurityProtocol());
> props.put("sasl.mechanism", "PLAIN"); // this could
> be different as per kafka acl policy. For our use case we fixed it to
> our implementation.
> }
> Post this lag is reported successfully on UI. SASL_MECHANISM property
> set is missing in the storm kafka monitor code base.
>
> Question -
> Now question here is that this is a standard problem and everyone
> must be facing it with an ACL enabled kafka cluster. So does storm
> have a standard fix for it? I navigated through
> TopologySpoutLag.class code and saw it can pass extra consumer
> properties through a temporary file creation but it doesn't work.
> Please let me know the solution around this.
>
> P.S. - I have not tried storm 2.5.0 yet but the code part does not
> have any change for this. I checked the master branch code and it is
> the same.
>
> Hope it answers all your questions. Please let me know if further
> info is required from me. Waiting for your response....
>
> Regards
> Sahil
>
>
>
>
> On Fri, Sep 29, 2023 at 7:34 PM Richard Zowalla <[email protected]>
> wrote:
> > Hi,
> >
> > think there is more context information required?
> >
> > - What exactly does "lagging" means?
> > - What is "the fix" in your custom JAR?
> > - Which modules are you using exactly?
> > - Does it also happen with 2.5.0 ?
> >
> > Gruß
> > Richard
> >
> > Am Freitag, dem 29.09.2023 um 08:55 +0530 schrieb Sahil Kamboj:
> > > Hi Team,
> > >
> > > Could you please help us with strom ui lag with acl enabled
> > > kafka.
> > > Currently we have to build a custom kafka monitor jar with a fix
> > > to
> > > get it worked properly. We are on version 2.4.0 and i can see in
> > > the
> > > code we do have a code snippet related to passing extra consumer
> > > configs but that is not working. Could you please let us know the
> > > correct way of doing this?
> > >
> > > PS: We have to set sasl.mechamism to the desired value in the
> > > props
> > > of consumer to get it worked under KafkaOffsetLagUtil.java
> > >
> > > Regards
> > > Sahil
> > >
> > > On Thu, 28 Sep, 2023, 5:46 pm Sahil Kamboj,
> > > <[email protected]> wrote:
> > > >
> > > >
> > > > On Thu, 28 Sep, 2023, 5:42 pm Sahil Kamboj,
> > > > <[email protected]> wrote:
> > > > > Hi Team,
> > > > >
> > > > > Could you please help us with strom ui lag with acl enabled
> > > > > kafka. Currently we have to build a custom kafka monitor jar
> > > > > with
> > > > > a fix to get it worked properly. We are on version 2.4.0 and
> > > > > i
> > > > > can see in the code we do have a code snippet related to
> > > > > passing
> > > > > extra consumer configs but that is not working. Could you
> > > > > please
> > > > > let us know the correct way of doing this?
> > > > >
> > > > > PS: We have to set sasl.mechamism to the desired value in the
> > > > > props of consumer to get it worked under
> > > > > KafkaOffsetLagUtil.java
> > > > >
> > > > > Regards
> > > > > Sahil
> >
