Re: error: ... protocols are incompatible with those of existing members ??
Thanks Dana, But this error is for a *single* process using the kafka-connect library. It does not seem to make any sense that the kafka-connect library would connect to the broker with two different protocols in the same process. Am I misunderstanding something? On Fri, Jun 10, 2016 at 6:55 PM, Dana Powers wrote: > Barry - i believe the error refers to the consumer group "protocol" that is > used to decide which partitions get assigned to which consumers. The way it > works is that each consumer says it wants to join X group and it can > support protocols (1, 2, 3...). The broker looks at all consumers in group > X and picks a protocol that all can support. If there is no common protocol > you would see this error. Example protocols are 'roundrobin' and 'range' . > > You should check the configuration of the group protocol for each consumer > and also check that you don't have extra consumers in the group, perhaps > because the group id is reused / common. > > Hope this helps, > > -Dana > On Jun 10, 2016 4:24 AM, "Barry Kaplan" wrote: > > I delete the group using kafka-consumer-groups.sh --delete and still I get > the error. >
Re: error: ... protocols are incompatible with those of existing members ??
Actually, this is exactly what Connect is doing. KafkaConnect uses its own "consumer" protocol called "connect" to distribute tasks between the workers. The default group name for this is connect-cluster, but it is possible to override it in the connect-distributed.properties file. SinkTasks also have a consumer (to get data from Kafka...), which uses the normal consumer protocol. The name for this group is "connect-" followed by the connector name. The connector name is configured with the connector itself (configuration file or REST). If you somehow managed to give both groups the same name, or (less expected) your connector has yet another consumer group in the plugin itself with the same name, you will see the error you reported. On Sat, Jun 11, 2016 at 10:16 AM, Barry Kaplan wrote: > Thanks Dana, > > But this error is for a *single* process using the kafka-connect library. > It does not seem to make any sense that the kafka-connect library would > connect to the broker with two different protocols in the same process. Am > I misunderstanding something? > > On Fri, Jun 10, 2016 at 6:55 PM, Dana Powers wrote: > >> Barry - i believe the error refers to the consumer group "protocol" that is >> used to decide which partitions get assigned to which consumers. The way it >> works is that each consumer says it wants to join X group and it can >> support protocols (1, 2, 3...). The broker looks at all consumers in group >> X and picks a protocol that all can support. If there is no common protocol >> you would see this error. Example protocols are 'roundrobin' and 'range' . >> >> You should check the configuration of the group protocol for each consumer >> and also check that you don't have extra consumers in the group, perhaps >> because the group id is reused / common. >> >> Hope this helps, >> >> -Dana >> On Jun 10, 2016 4:24 AM, "Barry Kaplan" wrote: >> >> I delete the group using kafka-consumer-groups.sh --delete and still I get >> the error. >>
Re: KafkaStream and Kafka consumer group
Do you instantiate KafkaProduer in your user code?
Why no use KStream.to("topic-name") ?
-Matthias
On 06/10/2016 12:28 AM, Saeed Ansari wrote:
> Thank you Eno,
> Adding more threads extremely increased the throughput of stream. As I said
> after processing I send the event to another topic. For that I was opening
> a connection via KafkaProducer to the cluster and I think that was the
> issue. Now there is just one producer for sending events to output topic.
>
> Do you have any recommendation how that part can get better?
>
> Thank you so much,
> Saeed
>
> On Thu, Jun 9, 2016 at 3:33 PM, Eno Thereska wrote:
>
>> Hi Saeed,
>>
>> There could be several reasons why things appear slow and it is difficult
>> to say without knowing the exact details of the setup and the results you
>> are observing.
>> One thing to check is the number of threads you have assigned to the Kafka
>> Stream application. By default just one thread is used. Perhaps you want
>> more (depending on number of cores you have). An example way to change that
>> in your app:
>>
>> props.put(StreamsConfig.NUM_STREAM_THREADS_CONFIG, 4 /* four threads */);
>>
>> Thanks
>> Eno
>>
>>> On 9 Jun 2016, at 18:08, Saeed Ansari wrote:
>>>
>>> Hi Eno,
>>> Thank you for the response. Actually I did not know it automatically
>>> assigns partitions to consumers. Now I have one Kafkastream reading from
>>> 12 partitions, like below:
>>> Controller is an actor that I am sending the message to and then it
>> creates
>>> child actors to send messages out.
>>>
>>> builder.stream("test").foreach((x, y) -> {
>>>controller.tell(y, controller.noSender());
>>> });
>>>
>>>
>>> The msg/sec rate I get from receiving messages to sending them out is
>>> really slow!
>>>
>>> Do you think it is about how consume messages?
>>>
>>> Thank you,
>>> Saeed
>>>
>>>
>>>
>>> On Wed, Jun 8, 2016 at 3:08 AM, Eno Thereska
>> wrote:
>>>
Hi Saeed,
Kafka Streams takes care of assigning partitions to consumers
automatically for you. You don't have to write anything explicit to do
that. See WordCountDemo.java as an example. Was there another reason you
wanted control over partition assignment?
Thanks
Eno
> On 7 Jun 2016, at 20:02, Saeed Ansari wrote:
>
> Hi,
> I have started a small Kafka streaming application. I need to assign
> partitions to consumers in a consumer group. I am not sure where to
>> start
> and how to structure consumer groups in KafkaStreams.
>
> I found that there is a StreamPartitionAssignor that can be added to
> config, but still I do not know how to configure it and what parameters
to
> config.
>
> Any help is really appreciated.
>
> Thank you,
> Saeed
>>
>>
>
signature.asc
Description: OpenPGP digital signature
Subscription
I want to get subscribed to Kafka mailing list
Re: Subscription
Hi Sauradipta, You want to send this email to users-subscr...@kafka.apache.org instead. -brian > On Jun 11, 2016, at 2:35 AM, sauradipta dutta > wrote: > > I want to get subscribed to Kafka mailing list
Re: KafkaStream and Kafka consumer group
When I read from stream, I create AKKA actors for processing events. I am
not processing them in the stream, as the result
I do not have KStream to write into it. So I use KafkaConsumer instead.
On Sat, Jun 11, 2016 at 7:13 AM, Matthias J. Sax
wrote:
> Do you instantiate KafkaProduer in your user code?
>
> Why no use KStream.to("topic-name") ?
>
>
> -Matthias
>
> On 06/10/2016 12:28 AM, Saeed Ansari wrote:
> > Thank you Eno,
> > Adding more threads extremely increased the throughput of stream. As I
> said
> > after processing I send the event to another topic. For that I was
> opening
> > a connection via KafkaProducer to the cluster and I think that was the
> > issue. Now there is just one producer for sending events to output topic.
> >
> > Do you have any recommendation how that part can get better?
> >
> > Thank you so much,
> > Saeed
> >
> > On Thu, Jun 9, 2016 at 3:33 PM, Eno Thereska
> wrote:
> >
> >> Hi Saeed,
> >>
> >> There could be several reasons why things appear slow and it is
> difficult
> >> to say without knowing the exact details of the setup and the results
> you
> >> are observing.
> >> One thing to check is the number of threads you have assigned to the
> Kafka
> >> Stream application. By default just one thread is used. Perhaps you want
> >> more (depending on number of cores you have). An example way to change
> that
> >> in your app:
> >>
> >> props.put(StreamsConfig.NUM_STREAM_THREADS_CONFIG, 4 /* four threads
> */);
> >>
> >> Thanks
> >> Eno
> >>
> >>> On 9 Jun 2016, at 18:08, Saeed Ansari wrote:
> >>>
> >>> Hi Eno,
> >>> Thank you for the response. Actually I did not know it automatically
> >>> assigns partitions to consumers. Now I have one Kafkastream reading
> from
> >>> 12 partitions, like below:
> >>> Controller is an actor that I am sending the message to and then it
> >> creates
> >>> child actors to send messages out.
> >>>
> >>> builder.stream("test").foreach((x, y) -> {
> >>>controller.tell(y, controller.noSender());
> >>> });
> >>>
> >>>
> >>> The msg/sec rate I get from receiving messages to sending them out is
> >>> really slow!
> >>>
> >>> Do you think it is about how consume messages?
> >>>
> >>> Thank you,
> >>> Saeed
> >>>
> >>>
> >>>
> >>> On Wed, Jun 8, 2016 at 3:08 AM, Eno Thereska
> >> wrote:
> >>>
> Hi Saeed,
>
> Kafka Streams takes care of assigning partitions to consumers
> automatically for you. You don't have to write anything explicit to do
> that. See WordCountDemo.java as an example. Was there another reason
> you
> wanted control over partition assignment?
>
> Thanks
> Eno
>
> > On 7 Jun 2016, at 20:02, Saeed Ansari
> wrote:
> >
> > Hi,
> > I have started a small Kafka streaming application. I need to assign
> > partitions to consumers in a consumer group. I am not sure where to
> >> start
> > and how to structure consumer groups in KafkaStreams.
> >
> > I found that there is a StreamPartitionAssignor that can be added to
> > config, but still I do not know how to configure it and what
> parameters
> to
> > config.
> >
> > Any help is really appreciated.
> >
> > Thank you,
> > Saeed
>
>
> >>
> >>
> >
>
>
Re: KafkaStream and Kafka consumer group
Sorry, I mean KafkaProducer!
On Sat, Jun 11, 2016 at 4:18 PM, Saeed Ansari
wrote:
> When I read from stream, I create AKKA actors for processing events. I am
> not processing them in the stream, as the result
> I do not have KStream to write into it. So I use KafkaConsumer instead.
>
> On Sat, Jun 11, 2016 at 7:13 AM, Matthias J. Sax
> wrote:
>
>> Do you instantiate KafkaProduer in your user code?
>>
>> Why no use KStream.to("topic-name") ?
>>
>>
>> -Matthias
>>
>> On 06/10/2016 12:28 AM, Saeed Ansari wrote:
>> > Thank you Eno,
>> > Adding more threads extremely increased the throughput of stream. As I
>> said
>> > after processing I send the event to another topic. For that I was
>> opening
>> > a connection via KafkaProducer to the cluster and I think that was the
>> > issue. Now there is just one producer for sending events to output
>> topic.
>> >
>> > Do you have any recommendation how that part can get better?
>> >
>> > Thank you so much,
>> > Saeed
>> >
>> > On Thu, Jun 9, 2016 at 3:33 PM, Eno Thereska
>> wrote:
>> >
>> >> Hi Saeed,
>> >>
>> >> There could be several reasons why things appear slow and it is
>> difficult
>> >> to say without knowing the exact details of the setup and the results
>> you
>> >> are observing.
>> >> One thing to check is the number of threads you have assigned to the
>> Kafka
>> >> Stream application. By default just one thread is used. Perhaps you
>> want
>> >> more (depending on number of cores you have). An example way to change
>> that
>> >> in your app:
>> >>
>> >> props.put(StreamsConfig.NUM_STREAM_THREADS_CONFIG, 4 /* four threads
>> */);
>> >>
>> >> Thanks
>> >> Eno
>> >>
>> >>> On 9 Jun 2016, at 18:08, Saeed Ansari wrote:
>> >>>
>> >>> Hi Eno,
>> >>> Thank you for the response. Actually I did not know it automatically
>> >>> assigns partitions to consumers. Now I have one Kafkastream reading
>> from
>> >>> 12 partitions, like below:
>> >>> Controller is an actor that I am sending the message to and then it
>> >> creates
>> >>> child actors to send messages out.
>> >>>
>> >>> builder.stream("test").foreach((x, y) -> {
>> >>>controller.tell(y, controller.noSender());
>> >>> });
>> >>>
>> >>>
>> >>> The msg/sec rate I get from receiving messages to sending them out is
>> >>> really slow!
>> >>>
>> >>> Do you think it is about how consume messages?
>> >>>
>> >>> Thank you,
>> >>> Saeed
>> >>>
>> >>>
>> >>>
>> >>> On Wed, Jun 8, 2016 at 3:08 AM, Eno Thereska
>> >> wrote:
>> >>>
>> Hi Saeed,
>>
>> Kafka Streams takes care of assigning partitions to consumers
>> automatically for you. You don't have to write anything explicit to
>> do
>> that. See WordCountDemo.java as an example. Was there another reason
>> you
>> wanted control over partition assignment?
>>
>> Thanks
>> Eno
>>
>> > On 7 Jun 2016, at 20:02, Saeed Ansari
>> wrote:
>> >
>> > Hi,
>> > I have started a small Kafka streaming application. I need to assign
>> > partitions to consumers in a consumer group. I am not sure where to
>> >> start
>> > and how to structure consumer groups in KafkaStreams.
>> >
>> > I found that there is a StreamPartitionAssignor that can be added to
>> > config, but still I do not know how to configure it and what
>> parameters
>> to
>> > config.
>> >
>> > Any help is really appreciated.
>> >
>> > Thank you,
>> > Saeed
>>
>>
>> >>
>> >>
>> >
>>
>>
>
storm mbeans
Hi, all I know kafka automatically expose mbeans to jmx, it seems storm doesn’t, i wonder if anyone has the experience to use JConsole to read storm build-in metrics through mbeans, or I will have to write separate metricConsumer to register metrics to mbeans? Is there such source code available? thanks AL
Introducing Dory
Hello Kafka users, Version 1.1.0 of Dory is now available. See https://github.com/dspeterson/dory for details. Dory is the successor to Bruce (https://github.com/tagged/bruce), a Kafka producer daemon I created while working at if(we) (http://www.ifwe.co/). The code has seen a number of improvements since its initial release in September 2014. The list of example clients for various programming languages has also been extended. Dory maintains full backward compatibility with Bruce, so existing users can easily switch. The latest release adds support for receiving messages from clients by UNIX domain stream socket or local TCP. Although UNIX domain datagrams are still the preferred means of sending messages in most cases, the option of using stream sockets facilitates sending messages too large to fit in a single datagram. The local TCP option facilitates adding support for clients written in programming languages that do not provide easy access to UNIX domain sockets. Dory's wiki page http://dory.wikidot.com/start contains a list of ideas for additional features and other improvements. Community feedback is welcomed and appreciated. If you have ideas for things you would like to see in future releases, please add them to the list. Also, please contribute code if you can afford the time. Thanks, Dave Peterson
Re: Regex topics in kafka connect?
Barry, Actually, it's not exposed in 0.10 either. https://issues.apache.org/jira/browse/KAFKA-3073 is filed to track this. We know people will want this, it just hasn't made it in quite yet. -Ewen On Fri, Jun 10, 2016 at 4:40 AM, Barry Kaplan wrote: > The kafka connect docs say > > The Kafka Connect framework manages any changes to the Kafka input, such as > > when the set of input topics changes because of a regex subscription. > > > But I can find no other information how to define a wildcard/regex topic > subscription. I tried giving the config topic property a regex but that > resulted in a error. I am working with 0.9, could it be that this is only > for 0.10? In any case, I still can't find the docs for regex topics in 0.10 > either. > > -barry > -- Thanks, Ewen
Broker Notification API for Kafka?
I'd like to add a notification API to Kafka so that I can receive the following notifications for each broker in the cluster: 1. An event notification when a broker is promoted from follower to leader, or when a follower catches up with the leader or falls behind the leader. 2. An event notification each time a new event is written to the leader partition. 3. An event notification each time an event is received by a partition follower. My motivation for having this API is two-fold: 1. I want to be able to do 'stream processing' without having to run a separate cluster for my app logic. 2. I want to keep the processing load next to the data, avoiding all unnecessary network communication. This makes sense for me because I want to use Kafka as an event-sourcing platform, rather than as a general streaming backbone between legacy applications. Could somebody with knowledge of the code base please provide me some pointers as to how I should approach adding an API like this, or provide further information if this is considered to be a bad idea. Thanks in advance, Dominic Chambers.
Re: Introducing Dory
Does Dory have the option to persist messages to local disk after the Dory ack but before sending to Kafka? Dave > On Jun 11, 2016, at 17:23, Dave Peterson wrote: > > Hello Kafka users, > > Version 1.1.0 of Dory is now available. See > https://github.com/dspeterson/dory for details. Dory is the successor > to Bruce (https://github.com/tagged/bruce), a Kafka producer daemon I > created while working at if(we) (http://www.ifwe.co/). The code has > seen a number of improvements since its initial release in September > 2014. The list of example clients for various programming languages > has also been extended. Dory maintains full backward compatibility > with Bruce, so existing users can easily switch. > > The latest release adds support for receiving messages from clients by > UNIX domain stream socket or local TCP. Although UNIX domain > datagrams are still the preferred means of sending messages in most > cases, the option of using stream sockets facilitates sending messages > too large to fit in a single datagram. The local TCP option > facilitates adding support for clients written in programming > languages that do not provide easy access to UNIX domain sockets. > > Dory's wiki page http://dory.wikidot.com/start contains a list of > ideas for additional features and other improvements. Community > feedback is welcomed and appreciated. If you have ideas for things > you would like to see in future releases, please add them to the list. > Also, please contribute code if you can afford the time. > > > Thanks, > Dave Peterson > > This e-mail and any files transmitted with it are confidential, may contain sensitive information, and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error, please notify the sender by reply e-mail immediately and destroy all copies of the e-mail and any attachments.
Re: Introducing Dory
There is an option for writing messages to a local log file, but this is intended purely for troubleshooting. Dory's intended purpose is to get messages off the local machine as quickly and efficiently as possible, and let the Kafka brokers handle the persistence. When a client sends a message to Dory, there is no ACK from Dory back to the client. An ACK isn't needed since Dory's clients are local, and the operating system guarantees the reliability of the interprocess communication. However, when Dory forwards the messages to Kafka, it waits for an ACK and resends messages as necessary due to communication failures or other errors. Once Dory has received a message from a client, it takes full responsibility for reliable delivery to Kafka. This simplifies clients, since sending messages becomes a simple one-way "fire it and forget it" communication process. Regards, Dave On 6/11/2016 7:50 PM, Tauzell, Dave wrote: Does Dory have the option to persist messages to local disk after the Dory ack but before sending to Kafka? Dave On Jun 11, 2016, at 17:23, Dave Peterson wrote: Hello Kafka users, Version 1.1.0 of Dory is now available. See https://github.com/dspeterson/dory for details. Dory is the successor to Bruce (https://github.com/tagged/bruce), a Kafka producer daemon I created while working at if(we) (http://www.ifwe.co/). The code has seen a number of improvements since its initial release in September 2014. The list of example clients for various programming languages has also been extended. Dory maintains full backward compatibility with Bruce, so existing users can easily switch. The latest release adds support for receiving messages from clients by UNIX domain stream socket or local TCP. Although UNIX domain datagrams are still the preferred means of sending messages in most cases, the option of using stream sockets facilitates sending messages too large to fit in a single datagram. The local TCP option facilitates adding support for clients written in programming languages that do not provide easy access to UNIX domain sockets. Dory's wiki page http://dory.wikidot.com/start contains a list of ideas for additional features and other improvements. Community feedback is welcomed and appreciated. If you have ideas for things you would like to see in future releases, please add them to the list. Also, please contribute code if you can afford the time. Thanks, Dave Peterson This e-mail and any files transmitted with it are confidential, may contain sensitive information, and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error, please notify the sender by reply e-mail immediately and destroy all copies of the e-mail and any attachments.
