Re: Any simple UI Tool for Kafka

[Rahul Singh]

Kafka Manager is good tool.

On Sat, Aug 24, 2019, 9:28 AM Darius Cooper  wrote:

> I'm looking for recommendations for a simple UI -based tool that will help
>
> * viewing lists of Kafka topics,
> * viewing Consumer groups for each topics
> * viewing messages for a topic
> * post test messages to a topic
>
> The audience would be developers who are using Kafka and could use a
> simple tool within their test environments.
>
> I found a couple of tools:
> * Kafka Tool
> * Yahoo Kafka-manager
>
> I'm looking for recommendations of more tools
>

Re: Kafka Consumer API - partition lags

[Rahul Singh]

Hi Garvit,

You can check here https://kafka.apache.org/documentation

Thanks,
Rahul

On Tue, Jun 25, 2019 at 4:11 PM Garvit Sharma  wrote:

> Hi All,
>
> I am looking for Kafka consumer API documentation to understand how it
> works internally.
>
> I am facing a problem where my consumer group is lagging behind very badly
> only on some partitions of the topic.
>
> Please let me know.
>
> Thanks,
>
>
> --
>
> Garvit Sharma
> github.com/garvitlnmiit/
>
> No Body is a Scholar by birth, its only hard work and strong determination
> that makes him master.
>

Error while creating topic using kafka-node client?

[Rahul Singh]

Hi All,

I am facing error while creating topic manually using kafka-node client.
The code is mentioned below.

Can anyone help please?

let topicsToCreate = [{ topic: topicName, partitions: 1, replicationFactor:
2 }];
admin.createTopics(topicsToCreate, (err, data) => {
if (err) {
console.log(err)
} else {
console.log(data)
}
});

The error it is throwing is given below ::

Error: No broker with id undefined
at KafkaClient.sendRequestToBroker
(/Kafka/KafkaDemo/node_modules/kafka-node/lib/kafkaClient.js:1089:21)
at getController
(/Kafka/KafkaDemo/node_modules/kafka-node/lib/kafkaClient.js:1190:10)
at loadMetadata
(/Kafka/KafkaDemo/node_modules/kafka-node/lib/kafkaClient.js:287:12)
at async.series
(/Kafka/KafkaDemo/node_modules/kafka-node/lib/kafkaClient.js:892:7)
at /Kafka/KafkaDemo/node_modules/async/dist/async.js:3888:9
at /Kafka/KafkaDemo/node_modules/async/dist/async.js:473:16
at replenish (/Kafka/KafkaDemo/node_modules/async/dist/async.js:1006:25)
at iterateeCallback
(/Kafka/KafkaDemo/node_modules/async/dist/async.js:995:17)
at /Kafka/KafkaDemo/node_modules/async/dist/async.js:969:16
at /Kafka/KafkaDemo/node_modules/async/dist/async.js:3885:13

Re: Max Limit of partitions in topic

[Rahul Singh]

There is no limit for partitioning in Kafka. It would be good the number of
partitions is equal to number of consumers. The consumer fetches a batch of
messages per partition. The more partitions that a consumer consumes, the
more memory it needs.

On Wed, Jan 23, 2019 at 12:25 PM marimuthu eee 
wrote:

> Hi,
>
> I have one dought, What is the maximum limit of partitions in one topic of
> kafka cluster.Please help me.
>

Re: How to acknowledge after consuming the message from Kafka topic?

[Rahul Singh]

Hi Daniel,

This is my code. Hopes it looks understandable, thanks :)

const kafka = require('kafka-node');
const ConsumerGroup = kafka.ConsumerGroup;

let options = {
kafkaHost: '127.0.0.1:9092',
groupId: 'DualTest',
autoCommit: false,
// autoCommitIntervalMs: 5000,
protocol: ['roundrobin'],
fromOffset: 'latest',
outOfRangeOffset: 'earliest',
sessionTimeout: 15000,
fetchMaxBytes: 10 * 1024 * 1024,
}
let consumerGroup = new ConsumerGroup(options, 'topicName')
consumerGroup.on('message', (message) => {
console.log(message);
consumerGroup.commit((err, res) => {
if (err) {
console.error(err)
} else {
console.log(res)
}
});
});

consumerGroup.on('error', (err) => {
console.log(`Error Occured ${err}`)
});

Here, the autoCommit property is set to false and committing manually by
consumerGroup.commit(), but when I restart the consumer it consumes all the
offsets from starting.

Thanks

On Mon, Jan 21, 2019 at 11:48 PM Daniel Hinojosa <
dhinoj...@evolutionnext.com> wrote:

> Show some code Rahul.
>
> On Mon, Jan 21, 2019 at 11:02 AM Rahul Singh <
> rahul.si...@smartsensesolutions.com> wrote:
>
> > I am using node-kafka, I have used consumer.commit to commit offsets but
> > don't know why when I restart the consumer it consume the committed
> > offsets.
> >
> > Thanks
> >
> > On Mon, Jan 21, 2019, 10:24 PM Hans Jespersen  >
> > > Are you using kafka-node or node-rdkafka? In either case you should
> call
> > > Consumer.commit(cb) or something similar to manually commit offsets
> (aka
> > > acknowledge messages).
> > >
> > > Alternatively so can set a config parameter on the consumer to
> > autoCommit.
> > >
> > > https://github.com/SOHU-Co/kafka-node/blob/master/README.md#consumer
> > >
> > > https://github.com/Blizzard/node-rdkafka/blob/master/README.md
> > >
> > > -hans
> > >
> > > > On Jan 21, 2019, at 5:17 AM, Rahul Singh <
> > > rahul.si...@smartsensesolutions.com> wrote:
> > > >
> > > > I am using in Node with node-kafka module.
> > > >
> > > >> On Mon, Jan 21, 2019 at 6:45 PM M. Manna 
> wrote:
> > > >>
> > > >> Please read KafkaConsumer javadoc - your answer is already there.
> > > >>
> > > >> Thanks,
> > > >>
> > > >> On Mon, 21 Jan 2019 at 13:13, Rahul Singh <
> > > >> rahul.si...@smartsensesolutions.com> wrote:
> > > >>
> > > >>> Hi All,
> > > >>>
> > > >>> I am testing kafka locally, I am able to produce and consume
> message.
> > > >> But,
> > > >>> after consuming the message from topic I want to acknowledge.
> > > >>>
> > > >>> Looking for solution. Please revert if anyone have.
> > > >>>
> > > >>> Thanks & Regards
> > > >>> Rahul Singh
> > > >>>
> > > >>
> > >
> >
>

Re: How to acknowledge after consuming the message from Kafka topic?

[Rahul Singh]

I am using node-kafka, I have used consumer.commit to commit offsets but
don't know why when I restart the consumer it consume the committed offsets.

Thanks

On Mon, Jan 21, 2019, 10:24 PM Hans Jespersen  Are you using kafka-node or node-rdkafka? In either case you should call
> Consumer.commit(cb) or something similar to manually commit offsets (aka
> acknowledge messages).
>
> Alternatively so can set a config parameter on the consumer to autoCommit.
>
> https://github.com/SOHU-Co/kafka-node/blob/master/README.md#consumer
>
> https://github.com/Blizzard/node-rdkafka/blob/master/README.md
>
> -hans
>
> > On Jan 21, 2019, at 5:17 AM, Rahul Singh <
> rahul.si...@smartsensesolutions.com> wrote:
> >
> > I am using in Node with node-kafka module.
> >
> >> On Mon, Jan 21, 2019 at 6:45 PM M. Manna  wrote:
> >>
> >> Please read KafkaConsumer javadoc - your answer is already there.
> >>
> >> Thanks,
> >>
> >> On Mon, 21 Jan 2019 at 13:13, Rahul Singh <
> >> rahul.si...@smartsensesolutions.com> wrote:
> >>
> >>> Hi All,
> >>>
> >>> I am testing kafka locally, I am able to produce and consume message.
> >> But,
> >>> after consuming the message from topic I want to acknowledge.
> >>>
> >>> Looking for solution. Please revert if anyone have.
> >>>
> >>> Thanks & Regards
> >>> Rahul Singh
> >>>
> >>
>

Re: How to acknowledge after consuming the message from Kafka topic?

[Rahul Singh]

I am using in Node with node-kafka module.

On Mon, Jan 21, 2019 at 6:45 PM M. Manna  wrote:

> Please read KafkaConsumer javadoc - your answer is already there.
>
> Thanks,
>
> On Mon, 21 Jan 2019 at 13:13, Rahul Singh <
> rahul.si...@smartsensesolutions.com> wrote:
>
> > Hi All,
> >
> > I am testing kafka locally, I am able to produce and consume message.
> But,
> > after consuming the message from topic I want to acknowledge.
> >
> > Looking for solution. Please revert if anyone have.
> >
> > Thanks & Regards
> > Rahul Singh
> >
>

How to acknowledge after consuming the message from Kafka topic?

[Rahul Singh]

Hi All,

I am testing kafka locally, I am able to produce and consume message. But,
after consuming the message from topic I want to acknowledge.

Looking for solution. Please revert if anyone have.

Thanks & Regards
Rahul Singh

Re: kafka cluster size planning

[Rahul Singh]

Why do you need that many partitions or topics — what’s the business use case.

Rahul Singh
Chief Executive Officer
m 202.905.2818

Anant Corporation
1010 Wisconsin Ave NW, Suite 250
Washington, D.C. 20007

We build and manage digital business technology platforms.
On Dec 10, 2018, 12:09 PM -0500, imamba , wrote:
> Hi, there,
> We will deploy a large-scale kafka cluster using kafka 2.0.0 in production 
> environment recently, as JunRao recommend in cluster-limit lately, to 
> accommodate for the rare event of a hard failure of the controller, it is 
> better to limit each broker to have up to 4,000 partitions and each cluster 
> to have up to 200,000 partitions. We plan to possess 2k topics each with 4k 
> partitions and two-replicas, it is clearly reaching the cluster-wide limit 
> recommended in above blog. Can such a large-scale kafka cluster meet our 
> production demand well?
>
>
> ps: broker hardware configuration described as follows:
> 24 cores
> 256G mem
> 10Gbps nic
> 22*4T sata disk
>
>
> Any advise/guidance would be greatly appreciated!
> Thanks!

Re: Kafka on Windows

[Rahul Singh]

I would recommend using Docker — it would end up being run on a Linux kernel VM 
on windows and is easier to get started on with a bit of learning curve for 
Docker. Less time wasted overall and at least at that point you would know 
Docker.

Rahul
On Aug 7, 2018, 4:50 AM -0400, jan , wrote:
> I tried using it just for learning a while back and wasted 3 days
> because it's not supported on windows. Even basic stuff didn't work. I
> did read the docs first!
>
> I think I've seen other people on this list have questions
> about/problems for exactly the same reason, and that could be a lot of
> time saved if it was in the docs - it needs to be. So how do I ask the
> maintainers to put 'No, Not Windows" in there?
> Serious question.
>
> I resent losing 3 days of work because of essential missing info. It
> sounds like (compared to @M. Manna) that I got off lightly.
>
> So can we put a clear caveat in the documentation, please, right at the top?
>
> jan
>
> On 07/08/2018, M. Manna  wrote:
> > The answer is - Absolutely not. If you don’t have Linux rack, or Kubernetes
> > deployment -it will not work on Windows as guaranteed.
> >
> > I know this because I have tried to make it work for the past 1 year. File
> > handling always fails and crashes the cluster on Windows.
> >
> > Thanks,
> >
> >
> >
> > On Tue, 7 Aug 2018 at 01:54, Alew  wrote:
> >
> > > Hi!
> > >
> > > Is it recommended to use production Kafka cluster on Windows?
> > >
> > > Can't get it from the documentation. It is possible to start Kafka on
> > > Windows, but maybe it's for development purposes only.
> > >
> > > Thanks.
> > >
> > >
> > >
> > >
> >

Re: Visualizing kafka topics data

[Rahul Singh]

Graylog, or Kibana (connected to ElasticSearch)


Rahul
On Aug 3, 2018, 5:12 PM -0400, Akash Jain , wrote:
> In my Java application, I am using a third party library to collect the
> memory profile, and send to a Kafka topic as JSON data.
>
> How can I make sense out of that JSON data? Is there a tool which can
> ingest data from a kafka topic and visualize it?
>
> Thanks,
> Akash

Re: Documentation/Article/Case Study on Scala as the Kafka Backbone Language

[Rahul Singh]

4. I meant to say pattern matching. You can catch a match on the structure 
(int, string, string) without explicitly setting the case class in a case 
switch statement. 
https://alvinalexander.com/scala/how-to-use-pattern-matching-scala-match-case-expressions

6. I’ve been using partial classes in c# since it came out 15 years ago. The 
partial classes / functions are awesome when having a part of the application 
generated by meta data and the other parts “baked”
Into the system. It’s a Deep subject , and primarily stylistic choice.

7. Javacc / antlr both generate a parser. Parboiled2 gives a DSL to define a 
parser In the native language. Parboiled1 had Java / Scalia. Parboiled2 only 
Scala. Imagine defining a parser and interactively testing it in REPL. Better 
yet , imagine being able to dynamically create a parsing syntax from data , 
dynamically. Javacc, Antlr would
Require a generation / validation step before then using it in your 
application. Not a fair comparison.

I agree . Syntactic sugar can be expensive — but when results matter- human 
time is more important and expensive than computer time. There are two reasons 
to really optimize code : shitty programming with bad design or actual 
legitimate reasons to squeeze that 5 millisecond out of the core.

Being able to create solutions quickly in 20% of what it normally take that 
work 80% of the time is better than spending 80% of the time to account for the 
20% of edge cases.

Best ,

Rahul
On Jul 28, 2018, 9:12 AM -0400, jan , wrote:
> I'm not a scala expert and haven't touched it for 18 months, but with
> respect to Mr. Singh, I'd like to clarify or question a few of his
> points.
>
> 1. Statelessness is a tool; not an end in itself but a means to an
> end. As someone on HackerNews says, "control your state space or die",
> but the same guy is *not* saying remove it all. I've seen immutability
> overused. Sometimes a bit of state makes things both more
> comprehensible and faster.
>
> 2. I don't know, and 3. true,
>
> 4. @Rahul, I don't understand, can you clarify?
>
> 5. Largely true and a huge bonus when used appropriately, but it can
> be overused. Sometimes it seems emphasis on "helpful" syntactic
> formatting without asking whether it actually helps the programmer.
>
> 6. Sounds like you've had more experience with them than me! Perhaps I
> don't know how to use them appropriately. I may be missing a trick.
>
> 7. I wouldn't argue but I'd warn that some abstractions can be
> expensive and I suspect shapeless may be one. Also, for parsers may I
> suggest looking at ANTLR?
>
> Idiomatic scala code can be expensive *as curremtly implemented*. Just
> understand that cost by profiling, and de-idiomise in hot code as
> needed.
>
> It's a fab language.
>
> jan
>
> On 23/07/2018, Rahul Singh  wrote:
> > Not necessarily for Kafka, but you can definitely google “Java vs. Scala”
> > and find a variety of reasons . I did a study for a client and ultimately
> > here are the major reasons I found :
> >
> > 1. Functional programming language which leads itself to stateless systems
> > 2. Better / easier to use stream processing syntax (then at that time in
> > Java 8)
> > 3. REPL available to quickly test functionality interactively.
> > 4. Case classes which can be inferred with or without strongly typed cases.
> > 5. Abilty to quickly create DSLs that seem natural to developers
> > 6. Awesome partial function syntax
> > 7. My personal favorite — as I was using parboiled2 to build a parser —
> > libraries like shapeless
> >
> > Best
> >
> > --
> > Rahul Singh
> > rahul.si...@anant.us
> >
> > Anant Corporation
> > On Jul 23, 2018, 8:40 AM -0400, M. Manna , wrote:
> > > Hello,
> > >
> > > Is anyone aware of any links or website where I can find information/case
> > > study etc. to why Scala was the best choice for kafka design? I hope this
> > > is not too much of a "Naive" question since I have had a very humble
> > > introduction to Scala.
> > >
> > > I understand that Scala is considered where distributed/scalable systems
> > > need to be designed. Also, in some cases it reduces multiple complex
> > > statements to be formed using a single complex statements i.e. reduce
> > > incremental verbosity.
> > >
> > > So, as a person who has background in Java, but relatively novice in
> > > Scala,
> > > I wanted to understand whether a study document exists to document the
> > > choice?
> > >
> > > Regards,
> >

Re: Documentation/Article/Case Study on Scala as the Kafka Backbone Language

[Rahul Singh]

Not necessarily for Kafka, but you can definitely google “Java vs. Scala” and 
find a variety of reasons . I did a study for a client and ultimately here are 
the major reasons I found :

1. Functional programming language which leads itself to stateless systems
2. Better / easier to use stream processing syntax (then at that time in Java 8)
3. REPL available to quickly test functionality interactively.
4. Case classes which can be inferred with or without strongly typed cases.
5. Abilty to quickly create DSLs that seem natural to developers
6. Awesome partial function syntax
7. My personal favorite — as I was using parboiled2 to build a parser — 
libraries like shapeless

Best

--
Rahul Singh
rahul.si...@anant.us

Anant Corporation
On Jul 23, 2018, 8:40 AM -0400, M. Manna , wrote:
> Hello,
>
> Is anyone aware of any links or website where I can find information/case
> study etc. to why Scala was the best choice for kafka design? I hope this
> is not too much of a "Naive" question since I have had a very humble
> introduction to Scala.
>
> I understand that Scala is considered where distributed/scalable systems
> need to be designed. Also, in some cases it reduces multiple complex
> statements to be formed using a single complex statements i.e. reduce
> incremental verbosity.
>
> So, as a person who has background in Java, but relatively novice in Scala,
> I wanted to understand whether a study document exists to document the
> choice?
>
> Regards,

Re: Real time streaming as a microservice

[Rahul Singh]

Seems like you need to expose your port via docker run or docker-compose .

https://docs.docker.com/v17.09/engine/userguide/networking/default_network/binding/



--
Rahul Singh
rahul.si...@anant.us

Anant Corporation
On Jul 9, 2018, 2:21 PM -0500, Mich Talebzadeh , 
wrote:
> Hi,
>
> I have now successfully created a docker for RHEL75 as follows:
>
> [root@rhes75 ~]# docker ps -a
> CONTAINER ID IMAGE COMMAND
> CREATED STATUS PORTS NAMES
> 816f07de15b1 zookeeper "/docker-entrypoint.…" 2 hours
> ago Up 2 hours 2181/tcp, 2888/tcp, 3888/tcp
> dockerZooKeeper
> 8dd84a174834 ubuntu "bash" 6 hours
> ago Up 6 hours
> dockerZooKeeperKafka
>
> The first container is ready made for ZooKeeper that exposes the zookeeper
> client port etc.
>
> The second container is an ubuntu shell which I installed both zookeeper
> and Kafka on it. They are both running in container dockerZooKeeperKafka
>
>
> hduser@8dd84a174834: /home/hduser/dba/bin> jps
> 5715 Kafka
> 5647 QuorumPeerMain
>
> hduser@8dd84a174834: /home/hduser/dba/bin> netstat -plten
> (Not all processes could be identified, non-owned process info
> will not be shown, you would have to be root to see it all.)
> Active Internet connections (only servers)
> Proto Recv-Q Send-Q Local Address Foreign Address
> State User Inode PID/Program name
> tcp 0 0 0.0.0.0: 0.0.0.0:*
> LISTEN 1005 2865148 5715/java
> tcp 0 0 0.0.0.0:35312 0.0.0.0:*
> LISTEN 1005 2865147 5715/java
> tcp 0 0 0.0.0.0:34193 0.0.0.0:*
> LISTEN 1005 2865151 5715/java
> tcp 0 0 0.0.0.0:22 0.0.0.0:*
> LISTEN 0 2757032 -
> tcp 0 0 0.0.0.0:40803 0.0.0.0:*
> LISTEN 1005 2852821 5647/java
>
>
> *tcp 0 0 0.0.0.0:9092 <http://0.0.0.0:9092>
> 0.0.0.0:* LISTEN 1005 2873507
> 5715/javatcp 0 0 0.0.0.0:2181 <http://0.0.0.0:2181>
> 0.0.0.0:* LISTEN 1005 2852829 5647/java*tcp6
> 0 0 :::22 :::* LISTEN
> 0 2757034 -
>
> I have a gateway node that is connected to the host running the container.
> From within the container I can ssh to the gateway host *as both the
> gateway host and host running the container are on the same VLAN.*
>
>
> However, I cannot connect from gateway to the container. The container has
> this IP address
>
> root@8dd84a174834:~# ifconfig -a
> eth0: flags=4163 mtu 1500
> *inet 172.17.0.2 netmask 255.255.0.0 broadcast 172.17.255.255*
> ether 02:42:ac:11:00:02 txqueuelen 0 (Ethernet)
> RX packets 173015 bytes 3263068025 (3.2 GB)
> RX errors 0 dropped 0 overruns 0 frame 0
> TX packets 189400 bytes 13557709 (13.5 MB)
> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
>
> lo: flags=73 mtu 65536
> inet 127.0.0.1 netmask 255.0.0.0
> loop txqueuelen 1000 (Local Loopback)
> RX packets 8450 bytes 534805 (534.8 KB)
> RX errors 0 dropped 0 overruns 0 frame 0
> TX packets 8450 bytes 534805 (534.8 KB)
> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
>
>
>
> The interesting thing is that in order to publish streaming test data I
> need to be able to do something like below
>
>
> cat ${PRICES} | ${KAFKA_HOME}/bin/kafka-console-producer.sh --broker-list
> rhes75:9092,rhes564:9092,rhes75:9093,rhes564:9093,rhes75:9094,rhes564:9094
> --topic md
>
>
> That Kafka broker list --broker-list
> rhes75:9092,rhes564:9092,rhes75:9093,rhes564:9093,rhes75:9094,rhes564:9094
> needs to be replaced by :9092!
>
>
> So at this juncture I am wondering what type of network needs to be created
> as the container is running within another host.
>
>
> Thanks
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * 
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Sun, 8 Jul 2018 at 20:00, Martin Gainty  wrote:
>
> >
> >
> > 
> > From: Mich Talebzadeh 
> > Sent: Sunday, July 8, 2018 1:01 PM
> > To: users@kafka.apache.org
> > Subject: Re: Real time streaming as a microservice
> >
> > Thanks Martin.
> >
> > From an implementation point of view do we need to introduce docker for
> > each microservice? In other words does it have to be artefact --> contain
> > --> docker for this to be true microservice an

Use of maxTickMessages in kafka ConsumerGroup?

[Rahul Singh]

Hello All,

I am setting maxTickMessages = 1 in Kafka consumer group while consuming
the record from the topic. It is giving me 2 records. I am not getting why
it is giving me 1 extra record of the mentioned size. Whenever I increase
the number it gives me one extra record.

 Please reply if anyone has a solution.

Thanks,
Rahul Singh

Avro Schema Generation from Json/Csv/SQL for Kafka topics

[Rahul Singh]

I have a Kafka related challenge and hoping someone else has faced this or has 
some pointers. This is NOT a *schema registry* question, it is a question 
regarding the generation of schemas. I already know how I’m managing these 
schemas once they are created.

I need to manage potentially several hundred topics which are primarily sourced 
from sources in a relational database accessible via JDBC and there several 
hundred consumers which will subscribe to them.

There are always changes that happen to the relational schema and thus need to 
be made to the avro schema which is being used in the topic and the processors.


I have a few solutions in mind:

1. Use Spark-Avro from Databricks to load the tables into a dataframe and then 
write using avro format, which then I have as a starting point.

2. Use Avro-SQL from Landoop -- but not sure if I need to have an existing 
table or if I can just give it arbitrary SQL.

3. Use other tools such as csv to avro, json to avro, but for each I need to do 
some preprocessing to create JSON to Avro, etc.

4. Any other options?

Goal is to walk through the tables in the database, review the metadata and 
generate Avro schemas, which would then be versioned / managed elsewhere. If 
there are changes to the topic group in general, we'd be automatically 
deleting/ adding topics to Kafka. I just don't want to task the team with 
manualy creating these avro schemas / topics.

If I'm going about it completely outside of left field, let me know.

Best,

--
Rahul Singh
rahul.si...@anant.us

Anant Corporation

Re: Monitoring Kafka

[Rahul Singh]

Without JMX may be difficult.. why not install an agent and report to an 
external service like ELK or new Relic?

That’s long standing industry pattern.

Some reading.. and some tools in the readings.. these articles are opinionated 
towards the vendors that published them but its a starting point.

https://blog.serverdensity.com/how-to-monitor-kafka/
https://www.datadoghq.com/blog/monitoring-kafka-performance-metrics/


On Apr 21, 2018, 6:54 AM -0400, Raghu Arur , wrote:
> Hi,
>
> Is there a way to pull broker stats (like partitions its is managing, jvm
> info, state of the partitions, etc.) without using JMX. We are shipping
> kafka in a appliance and there are restrictions on the ports that are open
> for security reasons. Are there any known ways of monitoring the health of
> Kafka ?
>
> Thanks,
> Raghu.

Re: Consuming messages from topics based on keys

[Rahul Singh]

I think this can be done in two ways.

1. Kstream or Ktable filter in a topology.
2. Store data in a persistent store elsewhere and expose via API (like 
Cassandra)

--
Rahul Singh
rahul.si...@anant.us

Anant Corporation

On Apr 19, 2018, 7:07 AM -0500, joe_delbri...@denso-diam.com, wrote:
> I am trying to determine how our consumer will send data to a rest API.
> The topics are machine based topics meaning they store all the information
> about a specific machine in one topic. I then have keys that are
> identifying the type of information stored. Here are some examples:
>
> Topic: E347-8 Key: machine-fault
> Topic: E347-8 Key: good-part-count
>
> Currently in the rest end point I am running the key through an if/else
> and the deciding how to handle it. Is there a better way to handle this?
> Can I have my consumer listen for a topic / key pair? I was thinking I
> could layout my consumers to be code specific like this example
>
> Consumer 1:
> Topic: E347-8 Key: machine_fault
> Topic: 186_7 Key: machine_fault
>
> Consumer 2:
> Topic: E347-8 Key: good-part-count
> Topic: 186_7 Key: good-part-count
>
>
> Thanks,
>
>
>
>
>
>
>
>
> The information contained in and transmitted with this Email may be
> privileged, proprietary, confidential and protected from disclosure. No
> privilege is hereby intended to be waived. This Email is intended only for
> the person to whom it is addressed. If you are not the intended
> recipient/addressee, any use of the Email and/or its contents, including,
> but not limited to, dissemination, distribution or copying is strictly
> prohibited and may be unlawful, and you must not take any action in
> reliance on it. If you receive this Email in error, please immediately
> notify the sender and delete the original message and any copies of it
> from your computer system. We deny any liability for damages resulting
> from the use of this Email by the unintended recipient, including the
> recipient in error.
>