Re: Help please: Topics deleted, but Kafka continues to try and sync them with Zookeeper

Thanks again for the replies. VERY much appreciated. I'll check both /admin/delete_topics and /config/topics. Chris On Thu, Jul 20, 2017 at 9:22 PM, Carl Haferd wrote: > If delete normally works, there would hopefully be some log entries when it > fails. Are

Re: Help please: Topics deleted, but Kafka continues to try and sync them with Zookeeper

If delete normally works, there would hopefully be some log entries when it fails. Are there any unusual zookeeper entries in the /admin/delete_topics path or in the other /admin folders? Does the topic name still exist in zookeeper under /config/topics? If so, that should probably deleted as

Re: Help please: Topics deleted, but Kafka continues to try and sync them with Zookeeper

Delete is definitely there. The delete worked fine, based on the fact that there is nothing in Zookeeper, and that the controller reported that the delete was successful, it's just something seems to have gotten out of sync. delete.topic.enabled is true. I've successfully deleted topics in the

Re: Help please: Topics deleted, but Kafka continues to try and sync them with Zookeeper

I could be totally wrong, but I seem to recall that delete wasn't fully implemented in 0.8.x? On Fri, Jul 21, 2017 at 10:10 AM, Carl Haferd wrote: > Chris, > > You could first check to make sure that delete.topic.enable is true and try > deleting again if not. If

Re: Help please: Topics deleted, but Kafka continues to try and sync them with Zookeeper

Chris, You could first check to make sure that delete.topic.enable is true and try deleting again if not. If that doesn't work with 0.8.1.1 you might need to manually remove the topic's log files from the configured log.dirs folder on each broker in addition to removing the topic's zookeeper

Re: [DISCUSS] KIP-177 Consumer perf tool should count rebalance time

I agree with Jason that we are just adding a new field so the impacted parser tools maybe limited. This additional information would be very useful. Guozhang On Wed, Jul 19, 2017 at 11:57 AM, Jason Gustafson wrote: > Ismael, I debated that also, but the main point was to

Re: Kafka Streams: why aren't offsets being committed?

Hi Bill, > When you say "even if the application has not had data for a long time" do you have a rough idea of how long? Minutes, hours > What is the value of your "auto.offset.reset" configuration? I don't specify it explicitly, but the ConsumerConfig logs indicate "auto.offset.reset =

Re: Kafka Streams: why aren't offsets being committed?

Hi Dmitry, When you say "even if the application has not had data for a long time" do you have a rough idea of how long? What is the value of your "auto.offset.reset" configuration? Thanks, Bill On Thu, Jul 20, 2017 at 6:03 PM, Dmitry Minkovsky wrote: > My Streams

Re: Consumer throughput drop

Hi Ovidu, The see-saw behavior is inevitable with linux when you have concurrent reads and writes. However, tuning the following two settings may help achieve more stable performance (from Jay's link): > *dirty_ratio*Defines a percentage value. Writeout of dirty data begins > (via *pdflush*)

Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand

Hi Jason, Regarding your comment about the current limitation on the information returned for a consumer group, do you think it's worth expanding the API to return some additional info (e.g. generation id, group leader, ...)? Thanks. --Vahid From: Jason Gustafson To:

Kafka Streams: why aren't offsets being committed?

My Streams application is configured to commit offsets every 250ms: Properties streamsConfig = new Properties(); streamsConfig.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 250); However, every time I restart my application, records that have already been processed are

Re: Spring release using apache clients 11

Thanks Rajini! El dia 20 jul. 2017 18:41, "Rajini Sivaram" va escriure: > David, > > The release plans are here: https://github.com/spring- > projects/spring-kafka/ > milestone/20?closed=1 > > We have already included TX and headers support to the current M3 which is >

Re: Consumer throughput drop

Yes, I’m using Debian Jessie 2.6 installed on this hardware [1]. It is also my understanding that Kafka is based on system’s cache (Linux in this case) which is based on Clock-Pro for page replacement policy, doing complex things for general workloads. I will check the tuning parameters, but I

Re: Consumer throughput drop

I suspect this is on Linux right? The way Linux works is it uses a percent of memory to buffer new writes, at a certain point it thinks it has too much buffered data and it gives high priority to writing that out. The good news about this is that the writes are very linear, well layed out, and

Named Pipe and Kafka Producer

Hi I am using named pipe and reading from it using Java and sending events to Kafka Cluster. The std out of a process is `tee` ed to But I am observing data loss. I am yet to debug this issue. I was wondering if anybody has already interfaced name pipe for sending data to kafka and what are the

Help please: Topics deleted, but Kafka continues to try and sync them with Zookeeper

Hi all, I have a weird situation here. I have deleted a few topics on my 0.8.1.1 cluster (old, I know...). The deletes succeeded according to the controller.log: [2017-07-20 16:40:31,175] INFO [TopicChangeListener on Controller 1]: New topics: [Set()], deleted topics:

Re: Spring release using apache clients 11

David, The release plans are here: https://github.com/spring-projects/spring-kafka/ milestone/20?closed=1 We have already included TX and headers support to the current M3 which is planned just after the next SF 5.0 RC3, which is expected tomorrow. Regards, Rajini On Thu, Jul 20, 2017 at 5:01

Re: Kafka with Zookeeper behind AWS ELB

Hello Pradeep, thank you for sharing your experience, will certainly consider it. On Thu, Jul 20, 2017 at 9:29 AM, Pradeep Gollakota wrote: > Luigi, > > I strongly urge you to consider a 5 node ZK deployment. I've always done > that in the past for resiliency during

Re: Kafka with Zookeeper behind AWS ELB

Luigi, I strongly urge you to consider a 5 node ZK deployment. I've always done that in the past for resiliency during maintenance. In a 3 node cluster, you can only tolerate one "failure", so if you bring one node down for maintenance and another node crashes during said maintenance, your ZK

Re: Kafka with Zookeeper behind AWS ELB

Yes Andrey, you can use an ENI without EIP on AWS if you only want a private address. After some consideration, I think that growing the zookeeper cluster more than 3 nodes is really unlikely so I think that I will attach 3 ENI to 3 servers in autoscaling and I will configure Kafka in using this

Re: On rebalance consumer group start from old offset

Solved by kafka-5600 Le mar. 18 juil. 2017 18:51, Sabarish Sasidharan a écrit : > This is similar to a problem I am also grappling with. We store the > processed offset for each partition in state store. And after restarts we > see that sometimes the start offset that

Spring release using apache clients 11

Hi, somebody know if we will any spring integration/kafka release soon using apache clients 11?

Reminder: Please accept my invitation to join Advanced Spark and TensorFlow Meetup

Advanced Spark and TensorFlow Meetup Join Chris Fregly and 7,676 other Spark and TensorFlow Experts in San Francisco. Be the first to hear about upcoming Meetups. Spark and Deep Learning Experts digging deep into the internals of Spark Core, Spark SQL, DataFrames, Spark Streaming, MLlib,

Kafka Connect distributed mode rebalance

I'm seeing some behavior with the DistributedHerder that I am trying to understand. I'm working on setting up a cluster of kafka connect nodes and have a relatively large number of connectors to submit to it (392 connectors right now that will soon become over 1100). As for the deployment of it I

Re: Get data from old offsets

Did you try setting `auto.offset.reset` to "earliest" ? -Matthias On 7/18/17 8:44 PM, Yuri da Costa Gouveia wrote: > Hello, > I am having trouble to get the data from old offsets. I'm using the version > 0.10.2.1, and I need any assistance to recover this data. > This is my consumer class: > >

Re: DAG processing in Kafka Streams

Sameer, the optimization you describe applies to batch processing but not to stream processing. As you mentioned: "will traverse the data only once". This property is interesting in batch processing only, as it means that the data is only read from disk once and both map operations are applies

Re: IllegalStateException with custom state store ..

https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-WhydoIgetanIllegalStateExceptionwhenaccessingrecordmetadata? -Matthias On 7/1/17 8:13 PM, Debasish Ghosh wrote: > Just to give some more information, the ProcessorContext that gets passed > to the init method of the custom store has a

Re: Custer startup order WRT Zookeeper Quorum for Kafka

You'll need a ZK quorum established before brokers boot, for sure. On Thu, Jul 20, 2017 at 12:53 PM, M. Manna wrote: > Hello, > > This might be too obvious for some people, but just thinking out loud here. > > So we need a recommended 3 node cluster to achieve the 1 point

Custer startup order WRT Zookeeper Quorum for Kafka

Hello, This might be too obvious for some people, but just thinking out loud here. So we need a recommended 3 node cluster to achieve the 1 point failure model. I am trying to deploy a 3 node cluster (3 zks and 3 brokers) in Linux (or Even Windows, doesn't matter here). Under the circumstance

Re: Streams StateStore topic name/reuse

Hi, I have two questions: > 1°/ Is the format written on this topic easily readable using the same > Serde I use for the state store or does Streams change it in any way? > If it is a KeyValue Store then you can use your Serdes to read from the changelog. > 2°/ since the topic will be used by

Streams StateStore topic name/reuse

Hello everyone, I currently run a small Streams app which accumulates data in a state store and periodically erases it. I would like another application (not running Kafka Streams), to consume the Kafka topic which backs this State Store and sometimes take actions depending on the state (of course

Re: [DISCUSS] KIP-177 Consumer perf tool should count rebalance time

OK, sounds good. Let's just make sure we note this in the upgrade notes. Ismael On Wed, Jul 19, 2017 at 11:57 AM, Jason Gustafson wrote: > Ismael, I debated that also, but the main point was to make users aware of > the rebalance latency (with KIP-134 in mind). I'm guessing

Re: Kafka with Zookeeper behind AWS ELB

The problem with EIP it is a public ip. Another option is to have the secondary interface attached to the instance on start(or a bit later) with the private static ip, but we are investigating the possibility. On Wed 19. Jul 2017 at 23:38, Luigi Tagliamonte < luigi.tagliamont...@gmail.com> wrote:

Re: Consumer blocked when auth failed.

What log level do you have configured? You might bump up to DEBUG or TRACE and see if anything stands out. -Jason On Tue, Jul 18, 2017 at 7:59 PM, 陈江枫 wrote: > Hi, > > I was integrating Kafka with Spark, using DirectStream, when my > authentication fail, the stream just