Re: [DISCUSS] KIP-174 - Deprecate and remove internal converter configs in WorkerConfig

2018-01-05 Thread UMESH CHAUDHARY
Thanks Ewen, and apologies as I missed this too. I'll start vote for this soon and proceed for the next steps. On Fri, 5 Jan 2018 at 09:03 Ewen Cheslack-Postava wrote: > Sorry I lost track of this thread. If things are in good shape we should > probably vote on this and get the deprecation commi

Consumer client not able to receive messages when one of broker is pushed down in the cluster

2018-01-05 Thread rAhul
Hi, I have a Apache kafka cluster with 3 nodes(say 1,2,3) with replication factor of 3 and partitions as 3. When my producer client, consumer client and the cluster are running, able to transfer messages from producer to consumer without any issues. Now I stopped leader node say node 1 from the

Exception during topic deletion when Kafka is hosted in Docker in Windows.

2018-01-05 Thread Alex Galperin
Hi, I host Kafka in Docker container in Windows. I mounted volume for storing Kafka data log. When I try to delete topic, I receive the following error: ERROR Error while deleting test-0 in dir /var/lib/kafka. (kafka.server.LogDirFailureChannel) java.io.IOException: Failed to rename log directory

Re: Consumer client not able to receive messages when one of broker is pushed down in the cluster

2018-01-05 Thread Hans Jespersen
Check that your __consumer_offsets topic is also setup with replication factor of 3 and has In Sync Replicas. Often it gets setup first as a one node cluster with RF=1 and then when the cluster is expanded to 3 nodes the step to increase the replication factor of this topic gets missed. -hans

Re: Kafka Streams Avro SerDe version/id caching

2018-01-05 Thread Kristopher Kane
Just a follow up caching example for the DSL with emphasis on the consumer (deserializer): final KafkaAvroDeserializer kafkaAvroDeserializer = new KafkaAvroDeserializer( new CachedSchemaRegistryClient(config.getProperty(AbstractKafkaAvroSerDeConfig.SCHEMA_REGISTRY_URL_CONFIG), 1024)); fi

Re: Exception during topic deletion when Kafka is hosted in Docker in Windows.

2018-01-05 Thread Ted Yu
Which Kafka release are you using ? Most likely /var/lib/kafka/test-0 was still being referenced by some thread. There have been fixes in this area recently. Cheers On Fri, Jan 5, 2018 at 4:28 AM, Alex Galperin wrote: > Hi, > I host Kafka in Docker container in Windows. I mounted volume for s

Re: kafka 1.0.1?

2018-01-05 Thread Brett Rann
Is there a plan/eta for this? > On 19 Dec 2017, at 08:49, Ismael Juma wrote: > > Hi Maciek, > > I expect that 1.0.1 will be released some time in January. > > Ismael > > On Mon, Dec 18, 2017 at 10:42 AM, Maciek Próchniak wrote: > > > Hello, > > > > are there plans to release version 1.0.1?

Re: Streams - State store directory created automatically

2018-01-05 Thread Matthias J. Sax
We always create task directories -- this helps to cleanly hand over tasks and detect reassignment issues (via the .lock) The checkpoint file will be empty if you don't use any state. -Matthias On 1/4/18 7:15 PM, Ted Yu wrote: > Looks like the .checkpoint file was generated from this code in > P

Re: Streams - State store directory created automatically

2018-01-05 Thread Kristopher Kane
Thanks! On Fri, Jan 5, 2018 at 3:13 PM, Matthias J. Sax wrote: > We always create task directories -- this helps to cleanly hand over > tasks and detect reassignment issues (via the .lock) > > The checkpoint file will be empty if you don't use any state. > > -Matthias > > On 1/4/18 7:15 PM, Ted

Insanely long recovery time with Kafka 0.11.0.2

2018-01-05 Thread Vincent Rischmann
Hello, so I'm upgrading my brokers from 0.10.1.1 to 0.11.0.2 to fix this bug https://issues.apache.org/jira/browse/KAFKA-4523 Unfortunately while stopping one broker, it crashed exactly because of this bug. No big deal usually, except after restarting Kafka in 0.11.0.2 the recovery is taking a rea

Re: Insanely long recovery time with Kafka 0.11.0.2

2018-01-05 Thread Brett Rann
What do the broker logs say its doing during all that time? There are some consumer offset / log cleaner bugs which caused us similarly log delays. that was easily visible by watching the log cleaner activity in the logs, and in our monitoring of partition sizes watching them go down, along with I