Re: KafkaStream and Kafka consumer group

[Matthias J. Sax]

I see.

Nevertheless, you should be able to use multiple producers. If you
increase the number of tasks, you should be able to create an own
producer for each task and thus increasing the write throughput.

-Matthias


On 06/11/2016 10:19 PM, Saeed Ansari wrote:
> Sorry, I mean KafkaProducer!
> 
> On Sat, Jun 11, 2016 at 4:18 PM, Saeed Ansari 
> wrote:
> 
>> When I read from stream, I create AKKA actors for processing events. I am
>> not processing them in the stream, as the result
>> I do not have KStream to write into it.  So I use KafkaConsumer instead.
>>
>> On Sat, Jun 11, 2016 at 7:13 AM, Matthias J. Sax 
>> wrote:
>>
>>> Do you instantiate KafkaProduer in your user code?
>>>
>>> Why no use KStream.to("topic-name") ?
>>>
>>>
>>> -Matthias
>>>
>>> On 06/10/2016 12:28 AM, Saeed Ansari wrote:
 Thank you Eno,
 Adding more threads extremely increased the throughput of stream. As I
>>> said
 after processing I send the event to another topic. For that I was
>>> opening
 a connection via KafkaProducer to the cluster and I think that was the
 issue. Now there is just one producer for sending events to output
>>> topic.

 Do you have any recommendation how that part can get better?

 Thank you so much,
 Saeed

 On Thu, Jun 9, 2016 at 3:33 PM, Eno Thereska 
>>> wrote:

> Hi Saeed,
>
> There could be several reasons why things appear slow and it is
>>> difficult
> to say without knowing the exact details of the setup and the results
>>> you
> are observing.
> One thing to check is the number of threads you have assigned to the
>>> Kafka
> Stream application. By default just one thread is used. Perhaps you
>>> want
> more (depending on number of cores you have). An example way to change
>>> that
> in your app:
>
> props.put(StreamsConfig.NUM_STREAM_THREADS_CONFIG, 4 /* four threads
>>> */);
>
> Thanks
> Eno
>
>> On 9 Jun 2016, at 18:08, Saeed Ansari  wrote:
>>
>> Hi Eno,
>> Thank you for the response. Actually I did not know it automatically
>> assigns partitions to consumers. Now I have one  Kafkastream reading
>>> from
>> 12 partitions, like below:
>> Controller is an actor that I am sending the message to and then it
> creates
>> child actors to send messages out.
>>
>> builder.stream("test").foreach((x, y) -> {
>>controller.tell(y, controller.noSender());
>> });
>>
>>
>> The msg/sec rate I get from receiving messages to sending them out is
>> really slow!
>>
>> Do you think it is about how consume messages?
>>
>> Thank you,
>> Saeed
>>
>>
>>
>> On Wed, Jun 8, 2016 at 3:08 AM, Eno Thereska 
> wrote:
>>
>>> Hi Saeed,
>>>
>>> Kafka Streams takes care of assigning partitions to consumers
>>> automatically for you. You don't have to write anything explicit to
>>> do
>>> that. See WordCountDemo.java as an example. Was there another reason
>>> you
>>> wanted control over partition assignment?
>>>
>>> Thanks
>>> Eno
>>>
 On 7 Jun 2016, at 20:02, Saeed Ansari 
>>> wrote:

 Hi,
 I have started a small Kafka streaming application. I need to assign
 partitions to consumers in a consumer group. I am not sure where to
> start
 and how to structure consumer groups in KafkaStreams.

 I found that there is a StreamPartitionAssignor that can be added to
 config, but still I do not know how to configure it and what
>>> parameters
>>> to
 config.

 Any help is really appreciated.

 Thank you,
 Saeed
>>>
>>>
>
>

>>>
>>>
>>
> 



signature.asc
Description: OpenPGP digital signature

Re: KafkaStream and Kafka consumer group

[Saeed Ansari]

Sorry, I mean KafkaProducer!

On Sat, Jun 11, 2016 at 4:18 PM, Saeed Ansari 
wrote:

> When I read from stream, I create AKKA actors for processing events. I am
> not processing them in the stream, as the result
> I do not have KStream to write into it.  So I use KafkaConsumer instead.
>
> On Sat, Jun 11, 2016 at 7:13 AM, Matthias J. Sax 
> wrote:
>
>> Do you instantiate KafkaProduer in your user code?
>>
>> Why no use KStream.to("topic-name") ?
>>
>>
>> -Matthias
>>
>> On 06/10/2016 12:28 AM, Saeed Ansari wrote:
>> > Thank you Eno,
>> > Adding more threads extremely increased the throughput of stream. As I
>> said
>> > after processing I send the event to another topic. For that I was
>> opening
>> > a connection via KafkaProducer to the cluster and I think that was the
>> > issue. Now there is just one producer for sending events to output
>> topic.
>> >
>> > Do you have any recommendation how that part can get better?
>> >
>> > Thank you so much,
>> > Saeed
>> >
>> > On Thu, Jun 9, 2016 at 3:33 PM, Eno Thereska 
>> wrote:
>> >
>> >> Hi Saeed,
>> >>
>> >> There could be several reasons why things appear slow and it is
>> difficult
>> >> to say without knowing the exact details of the setup and the results
>> you
>> >> are observing.
>> >> One thing to check is the number of threads you have assigned to the
>> Kafka
>> >> Stream application. By default just one thread is used. Perhaps you
>> want
>> >> more (depending on number of cores you have). An example way to change
>> that
>> >> in your app:
>> >>
>> >> props.put(StreamsConfig.NUM_STREAM_THREADS_CONFIG, 4 /* four threads
>> */);
>> >>
>> >> Thanks
>> >> Eno
>> >>
>> >>> On 9 Jun 2016, at 18:08, Saeed Ansari  wrote:
>> >>>
>> >>> Hi Eno,
>> >>> Thank you for the response. Actually I did not know it automatically
>> >>> assigns partitions to consumers. Now I have one  Kafkastream reading
>> from
>> >>> 12 partitions, like below:
>> >>> Controller is an actor that I am sending the message to and then it
>> >> creates
>> >>> child actors to send messages out.
>> >>>
>> >>> builder.stream("test").foreach((x, y) -> {
>> >>>controller.tell(y, controller.noSender());
>> >>> });
>> >>>
>> >>>
>> >>> The msg/sec rate I get from receiving messages to sending them out is
>> >>> really slow!
>> >>>
>> >>> Do you think it is about how consume messages?
>> >>>
>> >>> Thank you,
>> >>> Saeed
>> >>>
>> >>>
>> >>>
>> >>> On Wed, Jun 8, 2016 at 3:08 AM, Eno Thereska 
>> >> wrote:
>> >>>
>>  Hi Saeed,
>> 
>>  Kafka Streams takes care of assigning partitions to consumers
>>  automatically for you. You don't have to write anything explicit to
>> do
>>  that. See WordCountDemo.java as an example. Was there another reason
>> you
>>  wanted control over partition assignment?
>> 
>>  Thanks
>>  Eno
>> 
>> > On 7 Jun 2016, at 20:02, Saeed Ansari 
>> wrote:
>> >
>> > Hi,
>> > I have started a small Kafka streaming application. I need to assign
>> > partitions to consumers in a consumer group. I am not sure where to
>> >> start
>> > and how to structure consumer groups in KafkaStreams.
>> >
>> > I found that there is a StreamPartitionAssignor that can be added to
>> > config, but still I do not know how to configure it and what
>> parameters
>>  to
>> > config.
>> >
>> > Any help is really appreciated.
>> >
>> > Thank you,
>> > Saeed
>> 
>> 
>> >>
>> >>
>> >
>>
>>
>

Re: KafkaStream and Kafka consumer group

[Saeed Ansari]

When I read from stream, I create AKKA actors for processing events. I am
not processing them in the stream, as the result
I do not have KStream to write into it.  So I use KafkaConsumer instead.

On Sat, Jun 11, 2016 at 7:13 AM, Matthias J. Sax 
wrote:

> Do you instantiate KafkaProduer in your user code?
>
> Why no use KStream.to("topic-name") ?
>
>
> -Matthias
>
> On 06/10/2016 12:28 AM, Saeed Ansari wrote:
> > Thank you Eno,
> > Adding more threads extremely increased the throughput of stream. As I
> said
> > after processing I send the event to another topic. For that I was
> opening
> > a connection via KafkaProducer to the cluster and I think that was the
> > issue. Now there is just one producer for sending events to output topic.
> >
> > Do you have any recommendation how that part can get better?
> >
> > Thank you so much,
> > Saeed
> >
> > On Thu, Jun 9, 2016 at 3:33 PM, Eno Thereska 
> wrote:
> >
> >> Hi Saeed,
> >>
> >> There could be several reasons why things appear slow and it is
> difficult
> >> to say without knowing the exact details of the setup and the results
> you
> >> are observing.
> >> One thing to check is the number of threads you have assigned to the
> Kafka
> >> Stream application. By default just one thread is used. Perhaps you want
> >> more (depending on number of cores you have). An example way to change
> that
> >> in your app:
> >>
> >> props.put(StreamsConfig.NUM_STREAM_THREADS_CONFIG, 4 /* four threads
> */);
> >>
> >> Thanks
> >> Eno
> >>
> >>> On 9 Jun 2016, at 18:08, Saeed Ansari  wrote:
> >>>
> >>> Hi Eno,
> >>> Thank you for the response. Actually I did not know it automatically
> >>> assigns partitions to consumers. Now I have one  Kafkastream reading
> from
> >>> 12 partitions, like below:
> >>> Controller is an actor that I am sending the message to and then it
> >> creates
> >>> child actors to send messages out.
> >>>
> >>> builder.stream("test").foreach((x, y) -> {
> >>>controller.tell(y, controller.noSender());
> >>> });
> >>>
> >>>
> >>> The msg/sec rate I get from receiving messages to sending them out is
> >>> really slow!
> >>>
> >>> Do you think it is about how consume messages?
> >>>
> >>> Thank you,
> >>> Saeed
> >>>
> >>>
> >>>
> >>> On Wed, Jun 8, 2016 at 3:08 AM, Eno Thereska 
> >> wrote:
> >>>
>  Hi Saeed,
> 
>  Kafka Streams takes care of assigning partitions to consumers
>  automatically for you. You don't have to write anything explicit to do
>  that. See WordCountDemo.java as an example. Was there another reason
> you
>  wanted control over partition assignment?
> 
>  Thanks
>  Eno
> 
> > On 7 Jun 2016, at 20:02, Saeed Ansari 
> wrote:
> >
> > Hi,
> > I have started a small Kafka streaming application. I need to assign
> > partitions to consumers in a consumer group. I am not sure where to
> >> start
> > and how to structure consumer groups in KafkaStreams.
> >
> > I found that there is a StreamPartitionAssignor that can be added to
> > config, but still I do not know how to configure it and what
> parameters
>  to
> > config.
> >
> > Any help is really appreciated.
> >
> > Thank you,
> > Saeed
> 
> 
> >>
> >>
> >
>
>

Re: KafkaStream and Kafka consumer group

[Matthias J. Sax]

Do you instantiate KafkaProduer in your user code?

Why no use KStream.to("topic-name") ?


-Matthias

On 06/10/2016 12:28 AM, Saeed Ansari wrote:
> Thank you Eno,
> Adding more threads extremely increased the throughput of stream. As I said
> after processing I send the event to another topic. For that I was opening
> a connection via KafkaProducer to the cluster and I think that was the
> issue. Now there is just one producer for sending events to output topic.
> 
> Do you have any recommendation how that part can get better?
> 
> Thank you so much,
> Saeed
> 
> On Thu, Jun 9, 2016 at 3:33 PM, Eno Thereska  wrote:
> 
>> Hi Saeed,
>>
>> There could be several reasons why things appear slow and it is difficult
>> to say without knowing the exact details of the setup and the results you
>> are observing.
>> One thing to check is the number of threads you have assigned to the Kafka
>> Stream application. By default just one thread is used. Perhaps you want
>> more (depending on number of cores you have). An example way to change that
>> in your app:
>>
>> props.put(StreamsConfig.NUM_STREAM_THREADS_CONFIG, 4 /* four threads */);
>>
>> Thanks
>> Eno
>>
>>> On 9 Jun 2016, at 18:08, Saeed Ansari  wrote:
>>>
>>> Hi Eno,
>>> Thank you for the response. Actually I did not know it automatically
>>> assigns partitions to consumers. Now I have one  Kafkastream reading from
>>> 12 partitions, like below:
>>> Controller is an actor that I am sending the message to and then it
>> creates
>>> child actors to send messages out.
>>>
>>> builder.stream("test").foreach((x, y) -> {
>>>controller.tell(y, controller.noSender());
>>> });
>>>
>>>
>>> The msg/sec rate I get from receiving messages to sending them out is
>>> really slow!
>>>
>>> Do you think it is about how consume messages?
>>>
>>> Thank you,
>>> Saeed
>>>
>>>
>>>
>>> On Wed, Jun 8, 2016 at 3:08 AM, Eno Thereska 
>> wrote:
>>>
 Hi Saeed,

 Kafka Streams takes care of assigning partitions to consumers
 automatically for you. You don't have to write anything explicit to do
 that. See WordCountDemo.java as an example. Was there another reason you
 wanted control over partition assignment?

 Thanks
 Eno

> On 7 Jun 2016, at 20:02, Saeed Ansari  wrote:
>
> Hi,
> I have started a small Kafka streaming application. I need to assign
> partitions to consumers in a consumer group. I am not sure where to
>> start
> and how to structure consumer groups in KafkaStreams.
>
> I found that there is a StreamPartitionAssignor that can be added to
> config, but still I do not know how to configure it and what parameters
 to
> config.
>
> Any help is really appreciated.
>
> Thank you,
> Saeed


>>
>>
> 



signature.asc
Description: OpenPGP digital signature

Re: KafkaStream and Kafka consumer group

[Saeed Ansari]

Thank you Eno,
Adding more threads extremely increased the throughput of stream. As I said
after processing I send the event to another topic. For that I was opening
a connection via KafkaProducer to the cluster and I think that was the
issue. Now there is just one producer for sending events to output topic.

Do you have any recommendation how that part can get better?

Thank you so much,
Saeed

On Thu, Jun 9, 2016 at 3:33 PM, Eno Thereska  wrote:

> Hi Saeed,
>
> There could be several reasons why things appear slow and it is difficult
> to say without knowing the exact details of the setup and the results you
> are observing.
> One thing to check is the number of threads you have assigned to the Kafka
> Stream application. By default just one thread is used. Perhaps you want
> more (depending on number of cores you have). An example way to change that
> in your app:
>
> props.put(StreamsConfig.NUM_STREAM_THREADS_CONFIG, 4 /* four threads */);
>
> Thanks
> Eno
>
> > On 9 Jun 2016, at 18:08, Saeed Ansari  wrote:
> >
> > Hi Eno,
> > Thank you for the response. Actually I did not know it automatically
> > assigns partitions to consumers. Now I have one  Kafkastream reading from
> > 12 partitions, like below:
> > Controller is an actor that I am sending the message to and then it
> creates
> > child actors to send messages out.
> >
> > builder.stream("test").foreach((x, y) -> {
> >controller.tell(y, controller.noSender());
> > });
> >
> >
> > The msg/sec rate I get from receiving messages to sending them out is
> > really slow!
> >
> > Do you think it is about how consume messages?
> >
> > Thank you,
> > Saeed
> >
> >
> >
> > On Wed, Jun 8, 2016 at 3:08 AM, Eno Thereska 
> wrote:
> >
> >> Hi Saeed,
> >>
> >> Kafka Streams takes care of assigning partitions to consumers
> >> automatically for you. You don't have to write anything explicit to do
> >> that. See WordCountDemo.java as an example. Was there another reason you
> >> wanted control over partition assignment?
> >>
> >> Thanks
> >> Eno
> >>
> >>> On 7 Jun 2016, at 20:02, Saeed Ansari  wrote:
> >>>
> >>> Hi,
> >>> I have started a small Kafka streaming application. I need to assign
> >>> partitions to consumers in a consumer group. I am not sure where to
> start
> >>> and how to structure consumer groups in KafkaStreams.
> >>>
> >>> I found that there is a StreamPartitionAssignor that can be added to
> >>> config, but still I do not know how to configure it and what parameters
> >> to
> >>> config.
> >>>
> >>> Any help is really appreciated.
> >>>
> >>> Thank you,
> >>> Saeed
> >>
> >>
>
>

Re: KafkaStream and Kafka consumer group

[Eno Thereska]

Hi Saeed,

There could be several reasons why things appear slow and it is difficult to 
say without knowing the exact details of the setup and the results you are 
observing.
One thing to check is the number of threads you have assigned to the Kafka 
Stream application. By default just one thread is used. Perhaps you want more 
(depending on number of cores you have). An example way to change that in your 
app:

props.put(StreamsConfig.NUM_STREAM_THREADS_CONFIG, 4 /* four threads */);

Thanks
Eno

> On 9 Jun 2016, at 18:08, Saeed Ansari  wrote:
> 
> Hi Eno,
> Thank you for the response. Actually I did not know it automatically
> assigns partitions to consumers. Now I have one  Kafkastream reading from
> 12 partitions, like below:
> Controller is an actor that I am sending the message to and then it creates
> child actors to send messages out.
> 
> builder.stream("test").foreach((x, y) -> {
>controller.tell(y, controller.noSender());
> });
> 
> 
> The msg/sec rate I get from receiving messages to sending them out is
> really slow!
> 
> Do you think it is about how consume messages?
> 
> Thank you,
> Saeed
> 
> 
> 
> On Wed, Jun 8, 2016 at 3:08 AM, Eno Thereska  wrote:
> 
>> Hi Saeed,
>> 
>> Kafka Streams takes care of assigning partitions to consumers
>> automatically for you. You don't have to write anything explicit to do
>> that. See WordCountDemo.java as an example. Was there another reason you
>> wanted control over partition assignment?
>> 
>> Thanks
>> Eno
>> 
>>> On 7 Jun 2016, at 20:02, Saeed Ansari  wrote:
>>> 
>>> Hi,
>>> I have started a small Kafka streaming application. I need to assign
>>> partitions to consumers in a consumer group. I am not sure where to start
>>> and how to structure consumer groups in KafkaStreams.
>>> 
>>> I found that there is a StreamPartitionAssignor that can be added to
>>> config, but still I do not know how to configure it and what parameters
>> to
>>> config.
>>> 
>>> Any help is really appreciated.
>>> 
>>> Thank you,
>>> Saeed
>> 
>> 

Re: KafkaStream and Kafka consumer group

[Saeed Ansari]

Hi Eno,
Thank you for the response. Actually I did not know it automatically
assigns partitions to consumers. Now I have one  Kafkastream reading from
12 partitions, like below:
Controller is an actor that I am sending the message to and then it creates
child actors to send messages out.

builder.stream("test").foreach((x, y) -> {
controller.tell(y, controller.noSender());
});


The msg/sec rate I get from receiving messages to sending them out is
really slow!

Do you think it is about how consume messages?

Thank you,
Saeed



On Wed, Jun 8, 2016 at 3:08 AM, Eno Thereska  wrote:

> Hi Saeed,
>
> Kafka Streams takes care of assigning partitions to consumers
> automatically for you. You don't have to write anything explicit to do
> that. See WordCountDemo.java as an example. Was there another reason you
> wanted control over partition assignment?
>
> Thanks
> Eno
>
> > On 7 Jun 2016, at 20:02, Saeed Ansari  wrote:
> >
> > Hi,
> > I have started a small Kafka streaming application. I need to assign
> > partitions to consumers in a consumer group. I am not sure where to start
> > and how to structure consumer groups in KafkaStreams.
> >
> > I found that there is a StreamPartitionAssignor that can be added to
> > config, but still I do not know how to configure it and what parameters
> to
> > config.
> >
> > Any help is really appreciated.
> >
> > Thank you,
> > Saeed
>
>

Re: KafkaStream and Kafka consumer group

[Eno Thereska]

Hi Saeed,

Kafka Streams takes care of assigning partitions to consumers automatically for 
you. You don't have to write anything explicit to do that. See 
WordCountDemo.java as an example. Was there another reason you wanted control 
over partition assignment?

Thanks
Eno

> On 7 Jun 2016, at 20:02, Saeed Ansari  wrote:
> 
> Hi,
> I have started a small Kafka streaming application. I need to assign
> partitions to consumers in a consumer group. I am not sure where to start
> and how to structure consumer groups in KafkaStreams.
> 
> I found that there is a StreamPartitionAssignor that can be added to
> config, but still I do not know how to configure it and what parameters to
> config.
> 
> Any help is really appreciated.
> 
> Thank you,
> Saeed

KafkaStream and Kafka consumer group

[Saeed Ansari]

Hi,
I have started a small Kafka streaming application. I need to assign
partitions to consumers in a consumer group. I am not sure where to start
and how to structure consumer groups in KafkaStreams.

I found that there is a StreamPartitionAssignor that can be added to
config, but still I do not know how to configure it and what parameters to
config.

Any help is really appreciated.

Thank you,
Saeed