Re: Java API to read metrics via JMX

[Raghav]

Thanks Boris. Is there a sample implementation in Java ?

thanks.

R

On Wed, Aug 8, 2018 at 7:10 PM, Boris Lublinsky <
boris.lublin...@lightbend.com> wrote:

> Its actually quite simple, unfortunately you have to read, and then write
> to TSDB.
> Enclosed is an example doing this and dumping to InfluxDB
>
>
>
> Boris Lublinsky
> FDP Architect
> boris.lublin...@lightbend.com
> https://www.lightbend.com/
>
> On Aug 8, 2018, at 8:46 PM, Raghav  wrote:
>
> Hi
>
> Is there any Java API available so that I can enable our Kafka cluster's
> JMX port, and consume metrics via JMX api, and dump to a time series
> database.
>
> I checked out jmxtrans, but currently it does not dump to TSDB (time series
> database).
>
> Thanks.
>
> R
>
>
>
>


-- 
Raghav

Re: Java API to read metrics via JMX

[Boris Lublinsky]

Yea, this is just a simple snippet of how to do this.Boker with JMX is how to get broker from zk

BrokerWithJMX.scala
Description: Binary data

Boris LublinskyFDP Architectboris.lublin...@lightbend.comhttps://www.lightbend.com/


On Aug 8, 2018, at 9:18 PM, Ted Yu  wrote:Boris:BrokerWithJMX is referenced but I didn't find the class source after abrief search.FYIOn Wed, Aug 8, 2018 at 7:10 PM Boris Lublinsky  wrote:Its actually quite simple, unfortunately you have to read, and then writeto TSDB.Enclosed is an example doing this and dumping to InfluxDBBoris LublinskyFDP Architectboris.lublin...@lightbend.comhttps://www.lightbend.com/On Aug 8, 2018, at 8:46 PM, Raghav  wrote:HiIs there any Java API available so that I can enable our Kafka cluster'sJMX port, and consume metrics via JMX api, and dump to a time seriesdatabase.I checked out jmxtrans, but currently it does not dump to TSDB (time seriesdatabase).Thanks.R

Re: Java API to read metrics via JMX

[Ted Yu]

Boris:
BrokerWithJMX is referenced but I didn't find the class source after a
brief search.

FYI

On Wed, Aug 8, 2018 at 7:10 PM Boris Lublinsky <
boris.lublin...@lightbend.com> wrote:

> Its actually quite simple, unfortunately you have to read, and then write
> to TSDB.
> Enclosed is an example doing this and dumping to InfluxDB
>
>
> Boris Lublinsky
> FDP Architect
> boris.lublin...@lightbend.com
> https://www.lightbend.com/
>
> On Aug 8, 2018, at 8:46 PM, Raghav  wrote:
>
> Hi
>
> Is there any Java API available so that I can enable our Kafka cluster's
> JMX port, and consume metrics via JMX api, and dump to a time series
> database.
>
> I checked out jmxtrans, but currently it does not dump to TSDB (time series
> database).
>
> Thanks.
>
> R
>
>
>

Re: Java API to read metrics via JMX

[Boris Lublinsky]

Its actually quite simple, unfortunately you have to read, and then write to TSDB.Enclosed is an example doing this and dumping to InfluxDB

JMXCollector.scala
Description: Binary data

Boris LublinskyFDP Architectboris.lublin...@lightbend.comhttps://www.lightbend.com/


On Aug 8, 2018, at 8:46 PM, Raghav  wrote:HiIs there any Java API available so that I can enable our Kafka cluster'sJMX port, and consume metrics via JMX api, and dump to a time seriesdatabase.I checked out jmxtrans, but currently it does not dump to TSDB (time seriesdatabase).Thanks.R

Java API to read metrics via JMX

[Raghav]

Hi

Is there any Java API available so that I can enable our Kafka cluster's
JMX port, and consume metrics via JMX api, and dump to a time series
database.

I checked out jmxtrans, but currently it does not dump to TSDB (time series
database).

Thanks.

R

Looking for help with a question on the consumer API

[Moiz Raja (moraja)]

Hi All,

I have an issue with the consumer getting kicked out of the group possibly due 
to other issues going on in the system. The issue is detailed here 
https://stackoverflow.com/questions/51754794/how-to-reinstate-a-kafka-consumer-which-has-been-kicked-out-of-the-group

Any help with this issue would be appreciated.

Regards,
-Moiz

Re: python-kafka does not work

[Oras Al-Kubaisi]

Do you use auto commit or you commit messages directly? Do you have another
node in the same consumer group?

On Wed, 8 Aug 2018 at 16:18, 随风 <401580...@qq.com> wrote:

> hi，i use python-kafka to consumer message，but a few days later，it is does
> not work ，it can not to poll message.i can not deal with it util restart
> client
>
>
> can you tell me this reason and method
>
>
> 发自我的iPhone

Re: Error reading field 'version' when execute kafka-consumer-groups.sh

[Ted Yu]

For the last error, please see KAFKA-6868

On Wed, Aug 8, 2018 at 8:18 AM 刘鹏  wrote:

> hi,
> I get an error when I run kafka-consumer-groups.sh.
>
>
> firstly,I run:
> ./kafka-consumer-groups.sh --command-config consumer.properties
> --bootstrap-server ip:port --list
>
> the result is:
> Note: This will not show information about old Zookeeper-based
> consumers.
> mygroup
>
> then,I run:
>./kafka-consumer-groups.sh --command-config consumer.properties
> --bootstrap-server ip:port --describe --group mygroup
>
>
>the output I get is:
>Note: This will not show information about old Zookeeper-based
> consumers.
>Error: Executing consumer group command failed due to Error reading
> field 'version': java.nio.BufferUnderflowException
>
>
>my kafka version is:2.11-1.1.0,zookeeper version is:3.4.6,I have a 5
> nodes zk cluster and a 5 node kafka brokers cluster,they are in the same 5
> machine.I am using sarama-cluster as kafka client in golang.
>
>
>How can I find more detail of the error? Is anyone have some idea when
> this error occur?
>
>
>thanks a lot.
>
>
>
>
> thank you
> liupeng
>
>

python-kafka does not work

[????]

hi??i use python-kafka to consumer message??but a few days later??it is does 
not work ??it can not to poll message.i can not deal with it util restart client


can you tell me this reason and method


iPhone

Error reading field 'version' when execute kafka-consumer-groups.sh

[刘鹏]

hi,
I get an error when I run kafka-consumer-groups.sh.


firstly,I run:
./kafka-consumer-groups.sh --command-config consumer.properties 
--bootstrap-server ip:port --list
   
the result is:
Note: This will not show information about old Zookeeper-based consumers.
mygroup

then,I run:
   ./kafka-consumer-groups.sh --command-config consumer.properties 
--bootstrap-server ip:port --describe --group mygroup


   the output I get is:
   Note: This will not show information about old Zookeeper-based consumers.
   Error: Executing consumer group command failed due to Error reading field 
'version': java.nio.BufferUnderflowException


   my kafka version is:2.11-1.1.0,zookeeper version is:3.4.6,I have a 5 nodes 
zk cluster and a 5 node kafka brokers cluster,they are in the same 5 machine.I 
am using sarama-cluster as kafka client in golang.


   How can I find more detail of the error? Is anyone have some idea when this 
error occur? 


   thanks a lot.




thank you
liupeng

Re: Kafka flow

[Ted Yu]

Have you looked at http://kafka.apache.org/documentation/ ?

Chapters 4 and 5.

FYI

On Wed, Aug 8, 2018 at 1:46 AM darekAsz  wrote:

> Hi,
> I need informations about workflow in kafka, I want to know what is going
> on when producer send message, as much details as possible about
> requests-responses. Can anyone tell me when I can find this?
> Best regards
>

Include Keys in FileStreamSink connector

[David Espinosa]

Hi all,

I'm trying to backup the whole content of an avro topic into a file, and
later restoring the Kafka topic from the file using Kafka Connect. I'm
using avroconverters from both key and value, but key is not included in
the dump file.

Somebody knows how to include keys using Kafka Connect? (I'm using Kafka
Connect in distributed mode).

Btw, this is the request that I'm using to create the connector:

{
 "name": "mytopic-connector",
 "config": {
   "connector.class":
"org.apache.kafka.connect.file.FileStreamSourceConnector",
   "tasks.max": "1",
   "topic": "mytopic",
   "file": "/data/backups/mytopic.txt",
   "key.converter": "io.confluent.connect.avro.AvroConverter",
   "value.converter": "io.confluent.connect.avro.AvroConverter",
   "key.converter.schema.registry.url": "http://schema-registry:8081;,
   "value.converter.schema.registry.url": "http://schema-registry:8081;
 }
} (editado)

Re: Kafka on Windows

[M. Manna]

Using docker is probably the best option.

The one thing I cannot confirm (and I haven't tried yet) is how the volumes
will behave when you are transferring the topic log offsets to your windows
drive. Hopefully, this will be fine
but do let us know if someone has tried.

Thanks,



On 7 August 2018 at 23:33, jan  wrote:

> @Jacob Sheck: It was 18 months ago so I don't recall but I was told
> clearly windows ain't supported when I reported problems.
>
> I don't know if my problems were down to my inexperience or my use of
> windows, but I basically assuming I was controlling for one variable
> (n00bish inexperience) but was informed I now had two (n00bness, and
> an unsupported platform).
>
> tl;dr if it doesn't work on X, we need to say so clearly. It's just...
> good manners, surely?
>
> cheers
>
> jan
>
>
> On 07/08/2018, M. Manna  wrote:
> > By fully broken, i mean not designed and tested to work on Windows.
> >
> > On Tue, 7 Aug 2018, 16:34 M. Manna,  wrote:
> >
> >> Log Cleaner functionality is fully broken... If you haven't tried that
> >> already.
> >>
> >>
> >>
> >> On 7 Aug 2018 4:24 pm, "Jacob Sheck"  wrote:
> >>
> >> While I agree that it is less frustrating to run Kafka on Linux, I am
> >> interested to hear what specific issues you are running into on windows?
> >>
> >>
> >> On Tue, Aug 7, 2018 at 9:42 AM jan 
> >> wrote:
> >>
> >> > This is an excellent suggestion and I intend to do so henceforth
> >> > (thanks!), but it would be an adjunct to my request rather than the
> >> > answer; it still needs to be made clear in the docs/faq that you
> >> > *can't* use windows directly.
> >> >
> >> > jan
> >> >
> >> > On 07/08/2018, Rahul Singh  wrote:
> >> > > I would recommend using Docker — it would end up being run on a
> Linux
> >> > kernel
> >> > > VM on windows and is easier to get started on with a bit of learning
> >> > curve
> >> > > for Docker. Less time wasted overall and at least at that point you
> >> would
> >> > > know Docker.
> >> > >
> >> > > Rahul
> >> > > On Aug 7, 2018, 4:50 AM -0400, jan  invalid>,
> >> > wrote:
> >> > >> I tried using it just for learning a while back and wasted 3 days
> >> > >> because it's not supported on windows. Even basic stuff didn't
> work.
> >> > >> I
> >> > >> did read the docs first!
> >> > >>
> >> > >> I think I've seen other people on this list have questions
> >> > >> about/problems for exactly the same reason, and that could be a lot
> >> > >> of
> >> > >> time saved if it was in the docs - it needs to be. So how do I ask
> >> > >> the
> >> > >> maintainers to put 'No, Not Windows" in there?
> >> > >> Serious question.
> >> > >>
> >> > >> I resent losing 3 days of work because of essential missing info.
> It
> >> > >> sounds like (compared to @M. Manna) that I got off lightly.
> >> > >>
> >> > >> So can we put a clear caveat in the documentation, please, right at
> >> the
> >> > >> top?
> >> > >>
> >> > >> jan
> >> > >>
> >> > >> On 07/08/2018, M. Manna  wrote:
> >> > >> > The answer is - Absolutely not. If you don’t have Linux rack, or
> >> > >> > Kubernetes
> >> > >> > deployment -it will not work on Windows as guaranteed.
> >> > >> >
> >> > >> > I know this because I have tried to make it work for the past 1
> >> year.
> >> > >> > File
> >> > >> > handling always fails and crashes the cluster on Windows.
> >> > >> >
> >> > >> > Thanks,
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> > On Tue, 7 Aug 2018 at 01:54, Alew  wrote:
> >> > >> >
> >> > >> > > Hi!
> >> > >> > >
> >> > >> > > Is it recommended to use production Kafka cluster on Windows?
> >> > >> > >
> >> > >> > > Can't get it from the documentation. It is possible to start
> >> > >> > > Kafka
> >> > on
> >> > >> > > Windows, but maybe it's for development purposes only.
> >> > >> > >
> >> > >> > > Thanks.
> >> > >> > >
> >> > >> > >
> >> > >> > >
> >> > >> > >
> >> > >> >
> >> > >
> >> >
> >>
> >>
> >>
> >
>

Kafka flow

[darekAsz]

Hi,
I need informations about workflow in kafka, I want to know what is going
on when producer send message, as much details as possible about
requests-responses. Can anyone tell me when I can find this?
Best regards