Re: [VOTE] 0.10.2.0 RC1

Hi Ian, thanks for reporting this. I had a look at the stack trace and code and the whole situation is quite confusing. The exception itself is expected but we have a try-catch-block that should swallow the exception and it should never bubble up: In AbstractTaskCreator.retryWithBackoff a

consumer metrics for Spark streaming

Hi, I'm trying to access Kafka consumer metrics for a Spark streaming job and all I'm getting in JConsole is metrics for FetchRequestAndResponseMetrics. There's no other metrics (specially ConsumerLag MaxLag). Any thoughts? Environment: - Kafka 0.10.0.2.5 - Spark 2.0.x.2.5 - Spark / kafka

Kafka 0.10 and ZK 3.4.8 dependency upgrade

We tested Kafka 0.9 with zookeeper_3.4.5+dfsg-1_all.deb on Ubuntu 14.04 Trusty and want to use stable release of kafka_2.11-0.10.1.1 (hope, this has no critical issues) to avoid known producer issues, but, apparently this needs an upgraded ZK 3.4.8+ https://kafka.apache.org/

Re: Table a KStream

Sounds good, thank you! Kind regards, Nick On 10 February 2017 at 22:59, Matthias J. Sax wrote: > I agree that the API can be improved and we are working on that. > > Btw: KStream#toTable() was already suggested in KIP-114 discussion: > >

Re: Table a KStream

I agree that the API can be improved and we are working on that. Btw: KStream#toTable() was already suggested in KIP-114 discussion: http://search-hadoop.com/m/Kafka/uyzND19QaLMqiR2e1?subj=Re+DISCUSS+KIP+114+KTable+materialization+and+improved+semantics However for now, you can only choose

Re: Table a KStream

To be honest, I don't think either of these options are very good. stream.to("some-other-topic"); builder.table("some-other-topic"); As explained here , if the underlying topic doesn't have cleanup.policy=compact, the

Re: Partitioning behavior of Kafka Streams without explicit StreamPartitioner

Well, I think what you're doing is unusual for sure. The Streams API is really about transforming streams of data from input to output... so therefore the API doesn't have an injection point like you're looking for. I'd say it's intentional (I'm just a user though). If I were in your shoes, I'd

Re: Partitioning behavior of Kafka Streams without explicit StreamPartitioner

> On Feb 10, 2017, at 1:09 PM, Mathieu Fenniak > wrote: > > Hey Steven, > > If you have one KStream, and you want to produce to a topic that is read by > another KStream, you'd use the ".through" method of the first KStream. > ".through" both outputs to a topic

Re: Partitioning behavior of Kafka Streams without explicit StreamPartitioner

Hey Steven, If you have one KStream, and you want to produce to a topic that is read by another KStream, you'd use the ".through" method of the first KStream. ".through" both outputs to a topic and returns a KStream that reads from that topic. (".to" just outputs to a topic) If you want to

Re: Partitioning behavior of Kafka Streams without explicit StreamPartitioner

So then I guess my problem really is that I am operating at two different levels of abstraction. How do I produce to a KStream? I could imagine a method: public void KStream.put(K, V, Callback?); but I don't see anything like that. Nor do the "QueryableStoreTypes" really seem like what I want

Re: Potential memory leak in rocksdb

Hi Sachin, We have 6 consumers per node, each managing multiple partitions. We see the memory growing at the start of the application. To get the memory snapshot, download jemalloc here https://github.com/jemalloc/jemalloc/releases, compile and install it using standard command with the option

Re: Kafka Connect - Unknown magic byte

(the `kafka-avro-console-consumer` works because it doesn't deserialize the key.) Nick On 10 February 2017 at 19:25, Nick DeCoursin wrote: > Finally, here's the problem: > > $ curl -X GET localhost:8081/subjects > ["test-value","test-key","test2-value"] > > I

Re: How does one deploy to consumers without causing re-balancing for real time use case?

Pradeep is right. close() will try and send out a LeaveGroupRequest while a kill -9 will not. On Fri, Feb 10, 2017 at 10:19 AM, Pradeep Gollakota wrote: > I believe if you're calling the .close() method on shutdown, then the > LeaveGroupRequest will be made. If you're

Re: Kafka Connect - Unknown magic byte

Finally, here's the problem: $ curl -X GET localhost:8081/subjects ["test-value","test-key","test2-value"] I serialize the key and value as Avro, but using `.to(Serdes.String(), testSpecificAvroSerde, "test2");` converts the key to type String, not Avro. As such, there's no test2-key schema, in

Re: How does one deploy to consumers without causing re-balancing for real time use case?

I believe if you're calling the .close() method on shutdown, then the LeaveGroupRequest will be made. If you're doing a kill -9, I'm not sure if that request will be made. On Fri, Feb 10, 2017 at 8:47 AM, Praveen wrote: > @Pradeep - I just read your thread, the 1hr pause

Is anyone running Kafka on CoreOS?

Hi, (This question is kinda Kafka related, but mostly CoreOS related, so sorry if this is the wrong place to ask this.) Is anyone running Kafka on CoreOS? We run Kafka in docker containers on CoreOS. CoreOS has an OS-update policy where they will automatically install new OS updates, during

Re: [VOTE] 0.10.2.0 RC1

Seeing the following failure when using multi-threaded streams Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]: org.apache.kafka.streams.errors.LockException: task [0_21] Failed to lock the state directory: /tmp/kafka-streams/text_pipeline_id/0_21 Feb 10 17:21:15 ip-172-31-137-57

Passing SSL client principal to custom JAAS module with SSL or SASL_SSL

I need to create a custom JAAS module for authentication but I need to pass client certificate credentials as the principal. SASL_SSL mode has support for a JAAS module but from looking at the source code there doesn't appear to be a way to pass SSL client credentials to the module. The only

[VOTE] 0.10.2.0 RC1

Hello Kafka users, developers and client-developers, This is RC1 for release of Apache Kafka 0.10.2.0. This is a minor version release of Apache Kafka. It includes 19 new KIPs. See the release notes and release plan (https://cwiki.apache.org/ confluence/display/KAFKA/Release+Plan+0.10.2.0) for

Re: How does one deploy to consumers without causing re-balancing for real time use case?

@Pradeep - I just read your thread, the 1hr pause was when all the consumers where shutdown simultaneously. I'm testing out rolling restart to get the actual numbers. The initial numbers are promising. `STOP (1) (1min later kicks off) -> REBALANCE -> START (1) -> REBALANCE (takes 1min to get a

Re: How does one deploy to consumers without causing re-balancing for real time use case?

I asked a similar question a while ago. There doesn't appear to be a way to not triggering the rebalance. But I'm not sure why it would be taking > 1hr in your case. For us it was pretty fast. https://www.mail-archive.com/users@kafka.apache.org/msg23925.html On Fri, Feb 10, 2017 at 4:28 AM,

RE: about producer and consumer api

You'll need to provide some details. At a minimum the error message that you are getting. -Dave -Original Message- From: VIVEK KUMAR MISHRA 13BIT0066 [mailto:vivekkumar.mishra2...@vit.ac.in] Sent: Friday, February 10, 2017 4:22 AM To: users@kafka.apache.org Subject: about producer and

Old producer slow/no recovery on broker failure

We ran into an incident a while back where one of our broker machines abruptly went down (AWS is fun). While the leadership transitions and so forth seemed to work correctly with the remaining brokers, our producers hung shortly thereafter. I should point out that we are using the old Scala

Re: Need help in understanding bunch of rocksdb errors on kafka_2.10-0.10.1.1

Hi Sachin, Streams apps can be configured with a rocksdb.config.setter, which is a class name that needs to implement the org.apache.kafka.streams.state.RocksDBConfigSetter interface, which can be used to reduce the memory utilization of RockDB. Here's an example class that trims it way down

Re: Getting CommitFailedException in 0.10.2.0 due to member id is not valid or unknown

Hi, Understood. Just need to figure out the cause of these frequent re-balances. Somehow it seems to be pointing to rocksdb, but need to debug more. The pressing issue now is, to not kill the thread if there are commit failed exception on partition revoked (we anyway catch this at consumer

Re: Getting CommitFailedException in 0.10.2.0 due to member id is not valid or unknown

On 10 February 2017 at 11:18, Sachin Mittal wrote: > The heartbeat exception while rebalancing is OK. However I had some > different scenario which I wanted to understand. > > Please check line 42428 of https://dl.dropboxusercontent.com/u/46450177/ >

Re: How does one deploy to consumers without causing re-balancing for real time use case?

Would be great to get some input on it. - Krzysztof Lesniewski On 06.02.2017 08:27, Praveen wrote: I have a 16 broker kafka cluster. There is a topic with 32 partitions containing real time data and on the other side, I have 32 boxes w/ 1 consumer reading from these partitions. Today our

Re: Kafka Connect - Unknown magic byte

It seems like a bug. Thanks, Nick On 9 February 2017 at 14:57, Nick DeCoursin wrote: > Hello, > > Here is a github repo with the failing case: https://github.com/decoursin/ > kafka-connect-test. > > I've tried other similar things and nothing seems to work. > >

Re: Getting CommitFailedException in 0.10.2.0 due to member id is not valid or unknown

The heartbeat exception while rebalancing is OK. However I had some different scenario which I wanted to understand. Please check line 42428 of https://dl.dropboxusercontent.com/u/46450177/ TestKafkaAdvice.StreamThread-1.log Attempt to heartbeat failed for group new-part-advice since member id

Re: Potential memory leak in rocksdb

Hi, We also seem to be facing some potential rocksdb issue when there are more than one partition state store created on a machine. It looks like rocksdb is spending too much time in disk i/o. Could you please tell under what case you get the issue and also after how long running the streams

about producer and consumer api

Hello sir, I am learning Kafka. I know how to run kafka producers and consumers using terminal but know i want to run by my terminal i have written my sampleProducer and consumer code in java but i am not able to run . would you please help me out . Kindly look into that matter. thanks and

Re: Getting CommitFailedException in 0.10.2.0 due to member id is not valid or unknown

Hi Sachin, The CommitFailedException are thrown because the group is rebalancing. You can see log messages like below happening before the commit failed exception: Attempt to heartbeat failed for group new-part-advice since it is rebalancing. It isn't clear from the logs why the rebalancing is

Messages not received (C/C++ client)

hi?? Messages are not received when the partition is not consumed for the first time. The auto.offset.reset is set to be largest. Repetition of steps are as follows?? 1.create a group; 2.start a producer, and send messages to a fixed partition, such as partition 0; 3.start a consumer