Re: Strange issue in Kafka Windows where Log directory getting increased day by day

[Abhit Kalsotra]

Any pointers guys ?


Abhi

On Sun, Nov 27, 2016 at 11:14 AM, Abhit Kalsotra  wrote:

> Dear *
>
> I am facing some issue in Kafka Windows, the Log directories size are keep
> on increasing day by day..
>
> Currently the traffic on my kafka set up is minimal , but if this log
> directory size does not get fixed then its going to be serious issue...
>
> My Kafka SetUp
>
>- 2 Windows Server
>- kafka_2.11-0.10.0.1 and zoo keeper 3.4.9
>- 6 kafka broker nodes running on 2 Windows servers.
>- 2 topics, with 50 partitions and replication factor of 3.
>And one topic retention time is *8640(24 hours)*, and other one is 
> *360
>(1 Hour)*
>
> But every day I see the log directories are getting increased, where as
> the traffic on my Kafka is the constant.
>
> Please let me know what would be the resolution for the same. Also when i
> see the Log directories e.g
>
>
>
>
>
> *C:\kafka_2.11-0.10.0.1\kafka_2.11-0.10.0.1kafka-logs04\xyz-47inside this
> I could see .index and.log
> file existed around 10 days ago date too.*
> Please let me know what needs to be done
>
> As other wise its going to a big bottle neck for me
>
> Jira issue * >*
> Abhi
> !wq
> --
> If you can't succeed, call it version 1.0
>



-- 
If you can't succeed, call it version 1.0

Re: 0.8 consumers compatibility with Kafka 0.10

[Vladimir Kroz]

Hi Vladi,

There is a good place to start:
http://kafka.apache.org/documentation.html#upgrade_10
As you can see there are a lot of nitty-gritty aspects, so the answer is:
 "it depends"

Cheers,
Vlad

On Sun, Nov 27, 2016 at 12:09 PM Vladi Feigin  wrote:

> Hi All,
>
> We are using low-level API consumer in Kaka 0.8.2
> The question : Is Kafka 0.10 backward compatible with this old consumer ?
> Or we have to rewrite consumers when moving to 0.10?
>
> Thank you,
> Vladi
>
> --
>
> --
> This message may contain confidential and/or privileged information.
> If you are not the addressee or authorized to receive this on behalf of the
> addressee you must not use, copy, disclose or take action based on this
> message or any information herein.
> If you have received this message in error, please advise the sender
> immediately by reply email and delete this message. Thank you.
>

Re: --group flag for console consumer

[Andrew Pennebaker]

 Vahid, I'd like to see your PR if you please!

Confluent is on the ball.

On Wed, Nov 16, 2016 at 5:55 PM Gwen Shapira  wrote:

Thank you, Vahid!

On Wed, Nov 16, 2016 at 1:53 PM, Vahid S Hashemian
 wrote:
> I'll open a JIRA.
>
> Andrew, let me know if you want to take over the implementation.
> Otherwise, I'd be happy to work on it.
>
> Thanks.
> --Vahid
>
>
>
>
> From:   Gwen Shapira 
> To: Users 
> Date:   11/16/2016 01:23 PM
> Subject:Re: --group flag for console consumer
>
>
>
> Makes sense to me. Do you want to contribute a pull request?
>
> On Wed, Nov 16, 2016 at 11:33 AM, Andrew Pennebaker
>  wrote:
>> Could the kafka-console-consumer shell script please get a --group
> 
>> flag?
>>
>> Loading configs from properties files is helpful, but a direct --group
> flag
>> would be a simpler user interface for this common use case.
>>
>>
>> --
>> Cheers,
>> Andrew
>
>
>
> --
> Gwen Shapira
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter | blog
>
>
>
>
>



--
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

0.8 consumers compatibility with Kafka 0.10

[Vladi Feigin]

Hi All,

We are using low-level API consumer in Kaka 0.8.2
The question : Is Kafka 0.10 backward compatible with this old consumer ?
Or we have to rewrite consumers when moving to 0.10?

Thank you,
Vladi

--

-- 
This message may contain confidential and/or privileged information. 
If you are not the addressee or authorized to receive this on behalf of the 
addressee you must not use, copy, disclose or take action based on this 
message or any information herein. 
If you have received this message in error, please advise the sender 
immediately by reply email and delete this message. Thank you.

Re: Spring Kafka is not working with Spring 3.0.5.RELEASE

[Tauzell, Dave]

So I'm guessing you need to use spring 4 and not 3 like you are using.

Dave

> On Nov 27, 2016, at 10:58, Prasad Dls  wrote:
>
> Thanks Tauzell,
>
> java.lang.NoClassDefFoundError:
> org/springframework/core/task/AsyncListenableTaskExecutor class is part of
> spring-core 4.x.x
>
> On Sun, Nov 27, 2016 at 8:41 AM, Tauzell, Dave > wrote:
>
>> It looks like you are missing a spring jar. Can you google to find out
>> which jar that class is in?
>>
>> Dave
>>
>>> On Nov 27, 2016, at 01:16, Prasad Dls  wrote:
>>>
>>> Hi users,
>>>
>>>
>>> My project is already developed with Spring 3.0.5.RELEASE, We are
>> planning
>>> to use Kafka for new requirements. I am trying to use spring-kafka (from
>>> 1.0.x to 1.1.x) but it is not supporting with Spring 3.0.5.RELEASE.
>>> Application is throwing "class not found exception
>>> org.springframework.core.task.AsyncListenableTaskExecutor"
>>>
>>> Please help me how can I use spring kafka with spring3.0.5 .
>>>
>>>
>>>
>>> Thanks
>>> Prasad
>> This e-mail and any files transmitted with it are confidential, may
>> contain sensitive information, and are intended solely for the use of the
>> individual or entity to whom they are addressed. If you have received this
>> e-mail in error, please notify the sender by reply e-mail immediately and
>> destroy all copies of the e-mail and any attachments.
>>
This e-mail and any files transmitted with it are confidential, may contain 
sensitive information, and are intended solely for the use of the individual or 
entity to whom they are addressed. If you have received this e-mail in error, 
please notify the sender by reply e-mail immediately and destroy all copies of 
the e-mail and any attachments.

Re: Spring Kafka is not working with Spring 3.0.5.RELEASE

[Prasad Dls]

Thanks Tauzell,

java.lang.NoClassDefFoundError:
org/springframework/core/task/AsyncListenableTaskExecutor class is part of
spring-core 4.x.x

On Sun, Nov 27, 2016 at 8:41 AM, Tauzell, Dave  wrote:

> It looks like you are missing a spring jar. Can you google to find out
> which jar that class is in?
>
> Dave
>
> > On Nov 27, 2016, at 01:16, Prasad Dls  wrote:
> >
> > Hi users,
> >
> >
> > My project is already developed with Spring 3.0.5.RELEASE, We are
> planning
> > to use Kafka for new requirements. I am trying to use spring-kafka (from
> > 1.0.x to 1.1.x) but it is not supporting with Spring 3.0.5.RELEASE.
> > Application is throwing "class not found exception
> > org.springframework.core.task.AsyncListenableTaskExecutor"
> >
> > Please help me how can I use spring kafka with spring3.0.5 .
> >
> >
> >
> > Thanks
> > Prasad
> This e-mail and any files transmitted with it are confidential, may
> contain sensitive information, and are intended solely for the use of the
> individual or entity to whom they are addressed. If you have received this
> e-mail in error, please notify the sender by reply e-mail immediately and
> destroy all copies of the e-mail and any attachments.
>

Re: Spring Kafka is not working with Spring 3.0.5.RELEASE

[Tauzell, Dave]

It looks like you are missing a spring jar. Can you google to find out which 
jar that class is in?

Dave

> On Nov 27, 2016, at 01:16, Prasad Dls  wrote:
>
> Hi users,
>
>
> My project is already developed with Spring 3.0.5.RELEASE, We are planning
> to use Kafka for new requirements. I am trying to use spring-kafka (from
> 1.0.x to 1.1.x) but it is not supporting with Spring 3.0.5.RELEASE.
> Application is throwing "class not found exception
> org.springframework.core.task.AsyncListenableTaskExecutor"
>
> Please help me how can I use spring kafka with spring3.0.5 .
>
>
>
> Thanks
> Prasad
This e-mail and any files transmitted with it are confidential, may contain 
sensitive information, and are intended solely for the use of the individual or 
entity to whom they are addressed. If you have received this e-mail in error, 
please notify the sender by reply e-mail immediately and destroy all copies of 
the e-mail and any attachments.

Re: sharing storage topics between different distributed connect clusters

[Koert Kuipers]

got it thanks! i was thinking mostly along the lines of KAFKA-3815
, which i like a lot. it
would mean creating a cluster per connector. but if it also means creating
3 topics per connector then that makes things a little less convenient.

On Sun, Nov 27, 2016 at 1:24 AM, Ewen Cheslack-Postava 
wrote:

> If you want independent clusters based on the same Kafka cluster, they need
> independent values for config/offset/status topics. The Kafka Connect
> framework doesn't provide its own sort of namespacing or anything, so if
> you used the same topics in the same cluster, the values between different
> Connect clusters targeted at the same Kafka cluster would become conflated.
>
> -Ewen
>
> On Sat, Nov 26, 2016 at 7:59 PM, Koert Kuipers  wrote:
>
> > if i were to run multiple distributed connect clusters (so with different
> > group.id) does each connect cluster need its own offset.storage.topic,
> > config.storage.topic and status.storage.topic? or can they safely be
> shared
> > between the clusters?
> >
> > thanks!
> > koert
> >
>

Re: A strange controller log in Kafka 0.9.0.1

[Json Tu]

AnyBody？This is very disconcerting! If convenient, Can somebody help to confirm 
this strange question.

> 在 2016年11月26日，上午1:35，Json Tu  写道：
> 
> thanks guozhang,
>   if it's convenient，can we disscuss it in the jira 
> https://issues.apache.org/jira/browse/KAFKA-4447 
> ，I guess some body may also 
> encounter this problem.
> 
>> 在 2016年11月25日，下午12:31，Guozhang Wang  写道：
>> 
>> Does broker 100 keeps acting as the controller afterwards? What you observe
>> is possible and should be transient since "unsubscribeChildChanges" on
>> ZkClient and listener fired procedure are executed on different threads and
>> they are not strictly synchronized. But if you continuously see broker
>> 100's listener fires and it acts like a controller then there may be an
>> issue with 0.9.0.1 version.
>> 
>> Guozhang
>> 
>> On Wed, Nov 23, 2016 at 7:28 AM, Json Tu  wrote:
>> 
>>> Hi,
>>>   We have a cluster of kafka 0.9.0.1 with 3 nodes, and we found a
>>> strange controller log as below.
>>> 
>>> [2016-11-07 03:14:48,575] INFO [SessionExpirationListener on 100], ZK
>>> expired; shut down all controller components and try to re-elect
>>> (kafka.controller.KafkaController$SessionExpirationListener)
>>> [2016-11-07 03:14:48,578] DEBUG [Controller 100]: Controller resigning,
>>> broker id 100 (kafka.controller.KafkaController)
>>> [2016-11-07 03:14:48,579] DEBUG [Controller 100]: De-registering
>>> IsrChangeNotificationListener (kafka.controller.KafkaController)
>>> [2016-11-07 03:14:48,579] INFO [delete-topics-thread-100], Shutting down
>>> (kafka.controller.TopicDeletionManager$DeleteTopicsThread)
>>> [2016-11-07 03:14:48,579] INFO [delete-topics-thread-100], Stopped
>>> (kafka.controller.TopicDeletionManager$DeleteTopicsThread)
>>> [2016-11-07 03:14:48,579] INFO [delete-topics-thread-100], Shutdown
>>> completed (kafka.controller.TopicDeletionManager$DeleteTopicsThread)
>>> [2016-11-07 03:14:48,580] INFO [Partition state machine on Controller
>>> 100]: Stopped partition state machine (kafka.controller.
>>> PartitionStateMachine)
>>> [2016-11-07 03:14:48,580] INFO [Replica state machine on controller 100]:
>>> Stopped replica state machine (kafka.controller.ReplicaStateMachine)
>>> [2016-11-07 03:14:48,583] INFO [Controller-100-to-broker-101-send-thread],
>>> Shutting down (kafka.controller.RequestSendThread)
>>> [2016-11-07 03:14:48,584] INFO [Controller-100-to-broker-101-send-thread],
>>> Stopped  (kafka.controller.RequestSendThread)
>>> [2016-11-07 03:14:48,584] INFO [Controller-100-to-broker-101-send-thread],
>>> Shutdown completed (kafka.controller.RequestSendThread)
>>> [2016-11-07 03:14:48,586] INFO [Controller-100-to-broker-100-send-thread],
>>> Shutting down (kafka.controller.RequestSendThread)
>>> [2016-11-07 03:14:48,587] INFO [Controller-100-to-broker-100-send-thread],
>>> Stopped  (kafka.controller.RequestSendThread)
>>> [2016-11-07 03:14:48,587] INFO [Controller-100-to-broker-100-send-thread],
>>> Shutdown completed (kafka.controller.RequestSendThread)
>>> [2016-11-07 03:14:48,587] INFO [Controller 100]: Broker 100 resigned as
>>> the controller (kafka.controller.KafkaController)
>>> [2016-11-07 03:14:48,652] DEBUG [IsrChangeNotificationListener] Fired!!!
>>> (kafka.controller.IsrChangeNotificationListener)
>>> [2016-11-07 03:14:48,668] INFO [BrokerChangeListener on Controller 100]:
>>> Broker change listener fired for path /brokers/ids with children 101,100
>>> (kafka.controller.ReplicaStateMachine$BrokerChangeListener)
>>> [2016-11-07 03:14:48,683] DEBUG [DeleteTopicsListener on 100]: Delete
>>> topics listener fired for topics  to be deleted (kafka.controller.
>>> PartitionStateMachine$DeleteTopicsListener)
>>> [2016-11-07 03:14:48,687] INFO [AddPartitionsListener on 100]: Add
>>> Partition triggered {"version":1,"partitions":{"4"
>>> :[102,101],"5":[100,102],"1":[102,100],"0":[101,102],"2":[100,101],"3":[101,100]}}
>>> for path /brokers/topics/movie.gateway.merselllog.syncCinema
>>> (kafka.controller.PartitionStateMachine$AddPartitionsListener)
>>> [2016-11-07 03:14:48,694] INFO [AddPartitionsListener on 100]: Add
>>> Partition triggered {"version":1,"partitions":{"4"
>>> :[102,101],"5":[100,102],"1":[102,100],"0":[101,102],"2":[100,101],"3":[101,100]}}
>>> for path /brokers/topics/push_3rdparty_high (kafka.controller.
>>> PartitionStateMachine$AddPartitionsListener)
>>> [2016-11-07 03:14:48,707] INFO [AddPartitionsListener on 100]: Add
>>> Partition triggered {"version":1,"partitions":{"4"
>>> :[101,102],"5":[102,100],"1":[101,100],"0":[100,102],"2":[102,101],"3":[100,101]}}
>>> for path /brokers/topics/icb_msg_push_high_02 (kafka.controller.
>>> PartitionStateMachine$AddPartitionsListener)
>>> [2016-11-07 03:14:48,715] INFO [AddPartitionsListener on 100]: Add
>>> Partition triggered {"version":1,"partitions":{"4"
>>> :[102,100],"5":[100,101],"1":[102,101],"0":[101,100],"2":[100,102],"3":[101,102]}}
>>> for path /brokers/topics/movie.gateway.merselllog.u