Re: Kafka C/C++ client with consumer group support?

[Magnus Edenhill]

Hi Vince,

librdkafka (master branch) supports >=0.9 balanced consumers groups:
https://github.com/edenhill/librdkafka

As long as you are using different consumer groups each group will receive
the same set of messages.


Example C++ consumer:
https://github.com/edenhill/librdkafka/blob/master/examples/rdkafka_consumer_example.cpp#L398

API docs:
http://docs.confluent.io/2.0.1/clients/librdkafka/index.html


/Magnus

2016-04-06 16:16 GMT+02:00 Vince Deters :

> Hello, we have a requirement to run redundant Kafka consumers that consume
> an identical stream of messages in identical order. For this reason we plan
> to have these consumers consume from the same topic partition and identify
> themselves as unique consumer groups. Does any C/C++ Kafka client have this
> kind of consumer group support built in? Thanks,
> Vince
>

Re: Kafka C/C++ client with consumer group support?

[Ben Davison]

Hi Vince,

I think this might help you https://github.com/edenhill/librdkafka (

   - High-level balanced KafkaConsumer: supported (requires broker >= 0.9))

Regards,

Ben

On Wed, Apr 6, 2016 at 3:16 PM, Vince Deters  wrote:

> Hello, we have a requirement to run redundant Kafka consumers that consume
> an identical stream of messages in identical order. For this reason we plan
> to have these consumers consume from the same topic partition and identify
> themselves as unique consumer groups. Does any C/C++ Kafka client have this
> kind of consumer group support built in? Thanks,
> Vince
>

-- 


This email, including attachments, is private and confidential. If you have 
received this email in error please notify the sender and delete it from 
your system. Emails are not secure and may contain viruses. No liability 
can be accepted for viruses that might be transferred by this email or any 
attachment. Any unauthorised copying of this message or unauthorised 
distribution and publication of the information contained herein are 
prohibited.

7digital Limited. Registered office: 69 Wilson Street, London EC2A 2BB.
Registered in England and Wales. Registered No. 04843573.

RE: Kafka+in+C#

[Martin Gainty]

please be more specific on the kafka client you are implementing..here is the 
list:
https://cwiki.apache.org/confluence/display/KAFKA/Clients
Martin 
__ 




> Date: Mon, 28 Mar 2016 15:30:31 +0530
> Subject: Kafka+in+C#
> From: avadhootha...@gmail.com
> To: users@kafka.apache.org
> 
> We are starting a .Net project using Kafka, I have done a bit of research
> and found few repos that have c# clients. but I didn't find any
> documentation, how to use those clients programmatically. I was bit
> confused how to configure, Consume, Produce, commitoffset in my project.
> Can some help me on this, thanks in advance.
  

RE: Kafka in C#

[Thunder Stumpges]

Hi Matti,

We are actively testing in staging, under load and with extended reliability 
tests. It has been running well from a producer side for many days. We are 
pushing hard to get to production in the next couple of weeks however much of 
the testing is not directly related to the kafka client, but other changes in 
our core product.

As such, we have not had time to update documentation. Still, we would be 
pleased if you had a look and evaluated it; I think it is more fully featured 
than any of the other .net clients. If you do, please let me know how you get 
on. We'd be glad to help you out however we can.

Cheers,
Thunder


-Original Message-
From: Matti Waarna [mailto:mwaa...@sapient.com] 
Sent: Friday, January 09, 2015 9:43 AM
To: users@kafka.apache.org; Thunder Stumpges
Subject: RE: Kafka in C#

Thunder,

Have you had a chance to go to production with the C# producer and any updates 
on documentation?

I will be checking out the C# library sometime next week.

Thanks

-Matti

-Original Message-
From: Thunder Stumpges [mailto:tstump...@ntent.com]
Sent: Monday, December 22, 2014 1:10 PM
To: users@kafka.apache.org
Subject: RE: Kafka in C#

We definitely will. Wanted to wait until we got more confident the API was 
solid, ran in production for a little, and added a bit more documentation.

Cheers,
Thunder


-Original Message-
From: Joe Stein [joe.st...@stealth.ly]
Received: Monday, 22 Dec 2014, 10:05AM
To: users@kafka.apache.org [users@kafka.apache.org]
Subject: Re: Kafka in C#

Thunder, can you add that to
https://cwiki.apache.org/confluence/display/KAFKA/Clients#Clients-.net
didn't know it existed but cool that it uses Rx

On Mon, Dec 22, 2014 at 12:52 PM, Thunder Stumpges tstump...@ntent.com
wrote:

 Hi there,

 We looked at both of these a while back and ended up writing our own ( 
 https://github.com/ntent-ad/kafka4net).

 The first one in the list was completely synchronous, and had no 
 concept of batching. We initially attempted to use the second one 
 (kafka-net) but had some issues with detecting leader changes, 
 temporary errors, and general topic metadata changes. Also had some 
 problems with the use of async and the tracking of correlation IDs on tcp 
 messages.

 I know our client does not have a lot of documentation (yet) and says 
 Work in progress, not ready yet! but we have been testing this a 
 lot, and have been running in staging under load with good results. We 
 will be entering production in the next few weeks (waiting until after 
 the holidays). I wouldn't be comfortable with it going straight to 
 production without some testing in your environment, but especially 
 the Producer is very robust, and is resilient to all sorts of changes 
 in the environment (the integration tests use Vagrant and a set of VMs 
 and test Producing to non-existent partition and waiting for 
 auto-creation of topic, partition rebalancing, broker down or other 
 partition re-assignment, etc.) The client is fully Async and leverages 
 Rx (https://rx.codeplex.com/) and an event-loop-scheduler to do all 
 processing. Changes in partition state are broadcast Rx style to 
 listening components (Producer, Consumer, PartitionRecoveryMonitor).

 One of the reasons we have not finalized the documentation and notice 
 on the github page is we weren't sure if the API might change based on usage.
 To this end, we'd like to know what you think and if you have any 
 use-cases not handled by the API.

 I understand if you're not comfortable with the beta state of the 
 client, but we'd love to have you check it out. We are actively 
 developing on this and can help with any issues.

 Thanks,
 Thunder


 -Original Message-
 From: Matti Waarna [mailto:mwaa...@sapient.com]
 Sent: Monday, December 22, 2014 7:55 AM
 To: users@kafka.apache.org
 Subject: Kafka in C#

 We are using kafka version 0.8.1.1 and trying to produce from a C# app.

 I realize that there is no official C# library release and want to get 
 your experience with the existing solutions that are currently available.

 I am looking for a solution that is a) stable enough for production 
 environment and b) performs well.

 1) A couple of active github projects are available along with a few 
 forks each.
 Has Anybody worked on either of the following two options to 
 contribute their findings?
 https://github.com/Jroland/kafka-net

 https://github.com/miknil/Kafka4n

 2) Also there is the option of IKVM to import kafka JARS into a .net DLL.

 Maybe even another solution?

 Thanks

 MATTI

Re: Kafka in C#

[Joe Stein]

Another option is a HTTP wrapper around the actual producer and doing a
HTTP POST from C# to a REST server e.g.
https://github.com/stealthly/dropwizard-kafka-http which I know folks have
done successfully.

/***
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop
/

On Mon, Dec 22, 2014 at 10:55 AM, Matti Waarna mwaa...@sapient.com wrote:

 We are using kafka version 0.8.1.1 and trying to produce from a C# app.

 I realize that there is no official C# library release and want to get
 your experience with the existing solutions that are currently available.

 I am looking for a solution that is a) stable enough for production
 environment and b) performs well.

 1) A couple of active github projects are available along with a few forks
 each.
 Has Anybody worked on either of the following two options to contribute
 their findings?
 https://github.com/Jroland/kafka-net

 https://github.com/miknil/Kafka4n

 2) Also there is the option of IKVM to import kafka JARS into a .net DLL.

 Maybe even another solution?

 Thanks

 MATTI

RE: Kafka in C#

[Thunder Stumpges]

Hi there,

We looked at both of these a while back and ended up writing our own 
(https://github.com/ntent-ad/kafka4net). 

The first one in the list was completely synchronous, and had no concept of 
batching. We initially attempted to use the second one (kafka-net) but had some 
issues with detecting leader changes, temporary errors, and general topic 
metadata changes. Also had some problems with the use of async and the tracking 
of correlation IDs on tcp messages. 

I know our client does not have a lot of documentation (yet) and says Work in 
progress, not ready yet! but we have been testing this a lot, and have been 
running in staging under load with good results. We will be entering production 
in the next few weeks (waiting until after the holidays). I wouldn't be 
comfortable with it going straight to production without some testing in your 
environment, but especially the Producer is very robust, and is resilient to 
all sorts of changes in the environment (the integration tests use Vagrant and 
a set of VMs and test Producing to non-existent partition and waiting for 
auto-creation of topic, partition rebalancing, broker down or other partition 
re-assignment, etc.) The client is fully Async and leverages Rx 
(https://rx.codeplex.com/) and an event-loop-scheduler to do all processing. 
Changes in partition state are broadcast Rx style to listening components 
(Producer, Consumer, PartitionRecoveryMonitor). 

One of the reasons we have not finalized the documentation and notice on the 
github page is we weren't sure if the API might change based on usage. To this 
end, we'd like to know what you think and if you have any use-cases not handled 
by the API.

I understand if you're not comfortable with the beta state of the client, but 
we'd love to have you check it out. We are actively developing on this and can 
help with any issues.

Thanks,
Thunder


-Original Message-
From: Matti Waarna [mailto:mwaa...@sapient.com] 
Sent: Monday, December 22, 2014 7:55 AM
To: users@kafka.apache.org
Subject: Kafka in C#

We are using kafka version 0.8.1.1 and trying to produce from a C# app.

I realize that there is no official C# library release and want to get your 
experience with the existing solutions that are currently available.

I am looking for a solution that is a) stable enough for production environment 
and b) performs well.

1) A couple of active github projects are available along with a few forks each.
Has Anybody worked on either of the following two options to contribute their 
findings?
https://github.com/Jroland/kafka-net

https://github.com/miknil/Kafka4n

2) Also there is the option of IKVM to import kafka JARS into a .net DLL.

Maybe even another solution?

Thanks

MATTI

Re: Kafka in C#

[Joe Stein]

Thunder, can you add that to
https://cwiki.apache.org/confluence/display/KAFKA/Clients#Clients-.net
didn't know it existed but cool that it uses Rx

On Mon, Dec 22, 2014 at 12:52 PM, Thunder Stumpges tstump...@ntent.com
wrote:

 Hi there,

 We looked at both of these a while back and ended up writing our own (
 https://github.com/ntent-ad/kafka4net).

 The first one in the list was completely synchronous, and had no concept
 of batching. We initially attempted to use the second one (kafka-net) but
 had some issues with detecting leader changes, temporary errors, and
 general topic metadata changes. Also had some problems with the use of
 async and the tracking of correlation IDs on tcp messages.

 I know our client does not have a lot of documentation (yet) and says
 Work in progress, not ready yet! but we have been testing this a lot, and
 have been running in staging under load with good results. We will be
 entering production in the next few weeks (waiting until after the
 holidays). I wouldn't be comfortable with it going straight to production
 without some testing in your environment, but especially the Producer is
 very robust, and is resilient to all sorts of changes in the environment
 (the integration tests use Vagrant and a set of VMs and test Producing to
 non-existent partition and waiting for auto-creation of topic, partition
 rebalancing, broker down or other partition re-assignment, etc.) The client
 is fully Async and leverages Rx (https://rx.codeplex.com/) and an
 event-loop-scheduler to do all processing. Changes in partition state are
 broadcast Rx style to listening components (Producer, Consumer,
 PartitionRecoveryMonitor).

 One of the reasons we have not finalized the documentation and notice on
 the github page is we weren't sure if the API might change based on usage.
 To this end, we'd like to know what you think and if you have any use-cases
 not handled by the API.

 I understand if you're not comfortable with the beta state of the client,
 but we'd love to have you check it out. We are actively developing on this
 and can help with any issues.

 Thanks,
 Thunder


 -Original Message-
 From: Matti Waarna [mailto:mwaa...@sapient.com]
 Sent: Monday, December 22, 2014 7:55 AM
 To: users@kafka.apache.org
 Subject: Kafka in C#

 We are using kafka version 0.8.1.1 and trying to produce from a C# app.

 I realize that there is no official C# library release and want to get
 your experience with the existing solutions that are currently available.

 I am looking for a solution that is a) stable enough for production
 environment and b) performs well.

 1) A couple of active github projects are available along with a few forks
 each.
 Has Anybody worked on either of the following two options to contribute
 their findings?
 https://github.com/Jroland/kafka-net

 https://github.com/miknil/Kafka4n

 2) Also there is the option of IKVM to import kafka JARS into a .net DLL.

 Maybe even another solution?

 Thanks

 MATTI

RE: Kafka in C#

[Thunder Stumpges]

We definitely will. Wanted to wait until we got more confident the API was 
solid, ran in production for a little, and added a bit more documentation.

Cheers,
Thunder


-Original Message-
From: Joe Stein [joe.st...@stealth.ly]
Received: Monday, 22 Dec 2014, 10:05AM
To: users@kafka.apache.org [users@kafka.apache.org]
Subject: Re: Kafka in C#

Thunder, can you add that to
https://cwiki.apache.org/confluence/display/KAFKA/Clients#Clients-.net
didn't know it existed but cool that it uses Rx

On Mon, Dec 22, 2014 at 12:52 PM, Thunder Stumpges tstump...@ntent.com
wrote:

 Hi there,

 We looked at both of these a while back and ended up writing our own (
 https://github.com/ntent-ad/kafka4net).

 The first one in the list was completely synchronous, and had no concept
 of batching. We initially attempted to use the second one (kafka-net) but
 had some issues with detecting leader changes, temporary errors, and
 general topic metadata changes. Also had some problems with the use of
 async and the tracking of correlation IDs on tcp messages.

 I know our client does not have a lot of documentation (yet) and says
 Work in progress, not ready yet! but we have been testing this a lot, and
 have been running in staging under load with good results. We will be
 entering production in the next few weeks (waiting until after the
 holidays). I wouldn't be comfortable with it going straight to production
 without some testing in your environment, but especially the Producer is
 very robust, and is resilient to all sorts of changes in the environment
 (the integration tests use Vagrant and a set of VMs and test Producing to
 non-existent partition and waiting for auto-creation of topic, partition
 rebalancing, broker down or other partition re-assignment, etc.) The client
 is fully Async and leverages Rx (https://rx.codeplex.com/) and an
 event-loop-scheduler to do all processing. Changes in partition state are
 broadcast Rx style to listening components (Producer, Consumer,
 PartitionRecoveryMonitor).

 One of the reasons we have not finalized the documentation and notice on
 the github page is we weren't sure if the API might change based on usage.
 To this end, we'd like to know what you think and if you have any use-cases
 not handled by the API.

 I understand if you're not comfortable with the beta state of the client,
 but we'd love to have you check it out. We are actively developing on this
 and can help with any issues.

 Thanks,
 Thunder


 -Original Message-
 From: Matti Waarna [mailto:mwaa...@sapient.com]
 Sent: Monday, December 22, 2014 7:55 AM
 To: users@kafka.apache.org
 Subject: Kafka in C#

 We are using kafka version 0.8.1.1 and trying to produce from a C# app.

 I realize that there is no official C# library release and want to get
 your experience with the existing solutions that are currently available.

 I am looking for a solution that is a) stable enough for production
 environment and b) performs well.

 1) A couple of active github projects are available along with a few forks
 each.
 Has Anybody worked on either of the following two options to contribute
 their findings?
 https://github.com/Jroland/kafka-net

 https://github.com/miknil/Kafka4n

 2) Also there is the option of IKVM to import kafka JARS into a .net DLL.

 Maybe even another solution?

 Thanks

 MATTI