Re: Getting CommitFailedException in 0.10.2.0 due to member id is not valid or unknown

Hi, In continuation of the CommitFailedException what we observe is that when this happens first time ConsumerCoordinator invokes onPartitionsRevoked on StreamThread. This calls suspendTasksAndState() which again tries to commit offset and then again the same exception is thrown. This gets

Re: Kerberos/SASL Enabled Kafka - broker fails NoAuth due ACL

Hi, I used same principal and keytab across all brokers jass file ( Client section ) Still not working , now the second broker that starts is throwing 'Authentication failure' exception Do I need to set sasl.kerberos.principal.to.local.rules to something in all brokers ? On 09-Feb-2017 12:11

Re: Need help in understanding bunch of rocksdb errors on kafka_2.10-0.10.1.1

Hi, Please refer to thread Getting CommitFailedException in 0.10.2.0 due to member id is not valid or unknown I am adding more details to it. There are 2 questions 1. why we get CommitFailedException in case of member id unknown error returned by broker. 2. even if we get CommitFailedException

Re: Kafka Client 0.8.2.2 talks to Kafka Server 0.10.1.1

http://kafka.apache.org/081/documentation.html#highlevelconsumerapi On Thu, Feb 9, 2017 at 6:56 AM, Jeffrey Zhang wrote: > Manikumar, can you give a pointer where to find "old scala consumer API"? > thanks > Jeffrey > > On Wed, Feb 8, 2017 at 8:17 AM, Manikumar

Re: Kerberos/SASL Enabled Kafka - broker fails NoAuth due ACL

It is necessary to have the same principal name (in Client Section of jaas.config) across all brokers. Not sure why we need to modify kerberos.principal.to.local.rules in this case On Wed, Feb 8, 2017 at 11:48 PM, Ashish Bhushan wrote: > Hi , > > Were you able to resolve

Re: Need help in understanding bunch of rocksdb errors on kafka_2.10-0.10.1.1

Hi Sachin, It might be helpful if you send the logs from the streams application and the broker. Thanks, Damian On Thu, 9 Feb 2017 at 02:43, Sachin Mittal wrote: > Hi > I have upgraded to 2.10.2.0 however the streams is still falling via commit > failed exception with cause

Re: Kafka Streams: How to best maintain changelog indices using the DSL?

If the mapping is 1-to-1 than you can get it done. That's good. As you observed by yourself, with non-unique mapping it's way harder (or maybe even impossible) to get this. Also your KTable#groupBy(...)#aggregate(...) is a good solution. Thus, now I am just wondering, what you mean by: >

Re: KIP-121 [VOTE]: Add KStream peek method

+1 On 2/8/17 4:51 PM, Gwen Shapira wrote: > +1 (binding) > > On Wed, Feb 8, 2017 at 4:45 PM, Steven Schlansker > wrote: >> Hi everyone, >> >> Thank you for constructive feedback on KIP-121, >> KStream.peek(ForeachAction) ; >> it seems like it is time to call a

Re: Kafka Connect - Unknown magic byte

Any help here? I can create a git repo with the code, if somebody assures me they'll have a look. Thank you, Nick On 8 February 2017 at 10:39, Nick DeCoursin wrote: > Below's the rest of my consumer, which includes the serde code. It's worth > noting that when I run

Re: Need help in understanding bunch of rocksdb errors on kafka_2.10-0.10.1.1

Hi I have upgraded to 2.10.2.0 however the streams is still falling via commit failed exception with cause unknown member id. rocksdb locks issue sseem to be resolved. Also poll is called within the interval. I have posted more detail of the failure in another thread. Please let us know what

Re: Kafka Streams: How to best maintain changelog indices using the DSL?

Actually... I've got the 1-to-1 variant doing wonders for me. I replaced the #aggregate() with #reduce(((k, v) -> v, (k, v) -> null) and things are just lovely. Combining these indices with the various join operations, I am able to to build up deeply nested structures, or eh, materialized views,

Re: Kafka Client 0.8.2.2 talks to Kafka Server 0.10.1.1

Manikumar, can you give a pointer where to find "old scala consumer API"? thanks Jeffrey On Wed, Feb 8, 2017 at 8:17 AM, Manikumar wrote: > Are you using new java consumer API? It is officially released as part of > 0.9 release. > 0.8.2.2 java consumer code may not

Re: KIP-121 [VOTE]: Add KStream peek method

+1 (binding) On Wed, Feb 8, 2017 at 4:45 PM, Steven Schlansker wrote: > Hi everyone, > > Thank you for constructive feedback on KIP-121, KStream.peek(ForeachAction V>) ; > it seems like it is time to call a vote which I hope will pass easily :) > >

KIP-121 [VOTE]: Add KStream peek method

Hi everyone, Thank you for constructive feedback on KIP-121, KStream.peek(ForeachAction) ; it seems like it is time to call a vote which I hope will pass easily :) https://cwiki.apache.org/confluence/display/KAFKA/KIP-121%3A+Add+KStream+peek+method I believe the PR attached is already in

Re: KIP-122: Add a tool to Reset Consumer Group Offsets

Just to clarify, we'll need to allow specifying topic and partition. I don't think we want this on ALL partitions at once. On Wed, Feb 8, 2017 at 3:35 PM, Gwen Shapira wrote: > That's what I'd like to see. For example, suppose a Connect task fails > because it can't

Re: KIP-122: Add a tool to Reset Consumer Group Offsets

That's what I'd like to see. For example, suppose a Connect task fails because it can't deserialize an event from a partition. Stop connector, move offset forward, start connector. Boom! On Wed, Feb 8, 2017 at 3:22 PM, Matthias J. Sax wrote: > I am not sure about

Re: KIP-122: Add a tool to Reset Consumer Group Offsets

I am not sure about --reset-plus and --reset-minus Would this skip n messages forward/backward for each partitions? -Matthias On 2/8/17 2:23 PM, Jorge Esteban Quilcate Otoya wrote: > Great. I think I got the idea. What about this options: > > Scenarios: > > 1. Current status > >

Re: KIP-122: Add a tool to Reset Consumer Group Offsets

Great. I think I got the idea. What about this options: Scenarios: 1. Current status ´kafka-consumer-groups.sh --reset-offset --group cg1´ 2. To Datetime ´kafka-consumer-groups.sh --reset-offset --group cg1 --reset-to-datetime 2017-01-01T00:00:00.000´ 3. To Period ´kafka-consumer-groups.sh

Re: KIP-122: Add a tool to Reset Consumer Group Offsets

Hi, Just my few thoughts: does it need to be json? the old zkOffset tool had a nice format, very easy to manipulate on cli very powerfull: changes as many consumergroups/topics/partitions in one go as you want maybe allow -1 and -2 to indicate earliest and latest reset regardless of what

Re: Shutting down a Streams job

It is certainly possible, but when you got dozens of workers, that would take a very long time, specially if you got a lot of state, as partitions get reassigned and state moved about. In fact, it is likely to fail at some point, as local state that can be stored in a multitude of nodes may not

Re: KIP-122: Add a tool to Reset Consumer Group Offsets

Yes - using a tool like this to skip a set of consumer groups over a corrupt/bad message is definitely appealing. B On Wed, Feb 8, 2017 at 9:37 PM Gwen Shapira wrote: > I like the --reset-to-earliest and --reset-to-latest. In general, > since the JSON route is the most

Re: KIP-122: Add a tool to Reset Consumer Group Offsets

I like the --reset-to-earliest and --reset-to-latest. In general, since the JSON route is the most challenging for users, we want to provide a lot of ways to do useful things without going there. Two things that can help: 1. A lot of times, users want to skip few messages that cause issues and

Re: Kafka Streams: How to best maintain changelog indices using the DSL?

> And before we discuss deeper, a follow up question: if you map from to new_key, is this mapping "unique", or could it be that two different k/v-pairs map to the same new_key? Yes, this has been central in my exploration so far. For some fields the field is unique, for others it is not.

Re: Kafka Streams: How to best maintain changelog indices using the DSL?

It's difficult problem. And before we discuss deeper, a follow up question: if you map from to new_key, is this mapping "unique", or could it be that two different k/v-pairs map to the same new_key? If there are overlaps, you end up with a different problem as if there are no overlaps,

Re: Shutting down a Streams job

Can you take them down sequentially? Like, say, with a Kubernetes StatefulSet . On Wed, Feb 8, 2017 at 2:15 PM, Elias Levy wrote: > What are folks doing to

Re: KTable and cleanup.policy=compact

Yeah makes sense. I was looking at it from the point of view of keeping all data forever. Eno > On 8 Feb 2017, at 20:27, Matthias J. Sax wrote: > > Yes, that could happen if a key was not updated for a longer period than > topic retention time. > > If you want to force

Re: KTable and cleanup.policy=compact

Yes, that could happen if a key was not updated for a longer period than topic retention time. If you want to force a changelog creation, you can do a dummy aggregate instead of using KStreamBuilder#table() > KTable table = KStreamBuilder.stream("topic").groupByKey().reduce(new > Reducer() { >

Re: Kafka Client 0.8.2.2 talks to Kafka Server 0.10.1.1

Hi ismael, the reason is that we have our own Java package which sits on top of Kafka Java 0.8.2. I m new to Kafka and Scala. So to clarify, do I have to use Kafka scala client 0.8.2 in order to talk to server v0.10.1? If yes, where can I find the Kafka scala client? Thanks Jeffrey > On Feb 8,

Re: KTable and cleanup.policy=compact

I think there could be correctness implications... the default cleanup.policy of delete would mean that topic entries past the retention policy might have been removed. If you scale up the application, new application instances won't be able to restore a complete table into its local state store.

Re: Kafka Client 0.8.2.2 talks to Kafka Server 0.10.1.1

As Manikumar said, you need to use the Scala consumer if the client is v0.8.2.x. The Java consumer is only supported from 0.9.0.0. Is there a reason why you can't upgrade your client? Ismael On Wed, Feb 8, 2017 at 5:28 PM, Jeffrey Zhang wrote: > Hi Hans, thanks for

Re: KIP-121 [Discuss]: Add KStream peek method

Yes, thank you everyone for your input! I will incorporate the latest round of test revisions and submit a VOTE thread later today :) > On Feb 8, 2017, at 10:48 AM, Eno Thereska wrote: > > Steven, > > Sounds like we can start a VOTE thread on this? Is the KIP up to

Shutting down a Streams job

What are folks doing to cleanly shutdown a Streams job comprised of multiple workers? Right now I am doing sys.addShutdownHook(streams.close()) but that is not working well to shutdown a fleet of workers. When I signal the fleet to shutdown by sending them all a SIGTERM, some of them will

Re: KTable and cleanup.policy=compact

If you fail to set the policy to compact, there shouldn't be any correctness implications, however your topics will grow larger than necessary. Eno > On 8 Feb 2017, at 18:56, Jon Yeargers wrote: > > What are the ramifications of failing to do this? > > On Tue, Feb

Re: KTable and cleanup.policy=compact

What are the ramifications of failing to do this? On Tue, Feb 7, 2017 at 9:16 PM, Matthias J. Sax wrote: > Yes, that is correct. > > > -Matthias > > > On 2/7/17 6:39 PM, Mathieu Fenniak wrote: > > Hey kafka users, > > > > Is it correct that a Kafka topic that is used for

Re: KIP-121 [Discuss]: Add KStream peek method

Steven, Sounds like we can start a VOTE thread on this? Is the KIP up to date with all the latest comments? Thanks Eno > On 8 Feb 2017, at 18:05, Matthias J. Sax wrote: > > I like this idea. But to get clean and concise PRs, I would prefer to > have a JIRA and extra PR

Kafka Streams: How to best maintain changelog indices using the DSL?

I have a changelog that I'd like to index by some other key. So, something like this: class Item { byte[] id; String name; } KStreamBuilder topology = new KStreamBuilder(); KTable items = topology

Re: Kerberos/SASL Enabled Kafka - broker fails NoAuth due ACL

Hi , Were you able to resolve this problem ? On Fri, Jan 20, 2017 at 6:06 AM, amir masood khezrain < amir_li...@yahoo.com.invalid> wrote: > Hi > I am planning to setup a Kerberos/SASL enabled kafka cluster with three > brokers. Since “zookeeper.set.acl=true” is set, when running the first >

Re: KIP-121 [Discuss]: Add KStream peek method

I like this idea. But to get clean and concise PRs, I would prefer to have a JIRA and extra PR for this. WDYT? -Matthias On 2/8/17 9:35 AM, Guozhang Wang wrote: > The KIP proposal LGTM, thanks Steven! > > One meta comment on the PR itself: I'm wondering if we could refactoring > the

Re: Kafka Streams: Is automatic repartitioning before joins public/stable API?

Thank you. And thanks for the JIRA link. On Wed, Feb 8, 2017 at 12:22 AM, Matthias J. Sax wrote: > Yes, you can rely on this. > > The feature was introduced in Kafka 0.10.1 and will stay like this. We > already updated the JavaDocs (for upcoming 0.10.2, that is going to

Re: Potential memory leak in rocksdb

Well, I am a little perplexed now... I have already recompiled the branch 0.10.1 with rocksdb 4.11.2 and it doesn't seem better. So I have modified the launcher of our jvm to use jemalloc with the profile enabled and from the first result I have, it seems that the problem comes from the method

Re: KIP-121 [Discuss]: Add KStream peek method

The KIP proposal LGTM, thanks Steven! One meta comment on the PR itself: I'm wondering if we could refactoring the implementation of `KStream.print() / writeAsText()` to just be a special impl of `peek()` then, like we did for `count` as for `aggregate`? I.e. we can replace the `KeyValuePrinter`

Re: Reg: Reading consumer groups and lag

Yes, I can do that from java with something like, *ConsumerGroupCommand.main(new String[]{"--new-consumer", "--bootstrap-server", "localhost:9092","--describe", "--group", "testgroup"});* But how do i process the output as it does not return anything? Thanks, Sumit On Wed, Feb 8, 2017 at

Re: Potential memory leak in rocksdb

Hello Pierre, As Damian said your code looks fine and I cannot think of a direct reason for the rocksdb memory leak on top of my head. Could you build and try out the latest Kafka trunk (will be released as 0.10.2 in a few days) which contains a newer version of RocksDB and see if this issue

Re: Kafka Client 0.8.2.2 talks to Kafka Server 0.10.1.1

Hi Hans, thanks for your reply. Here is what I see - for the Kafka client v0.8.2.2, the consumer.poll method returns an object of Map while it returns ConsumerRecords from Kafka Server v0.10.1.1. that is why Kafka client v0.8.2.2 now does not work with my Kafka Kafka

Re: Reg: Reading consumer groups and lag

You can run the same class executed in the scripts. On Feb 8, 2017 8:50 AM, "Sumit Maheshwari" wrote: > Hi, > > Currently in 0.10 we can get the information about the consumer groups and > respective lag using the kafka-consumer-groups.sh. > Is there a way to achieve the

Reg: Reading consumer groups and lag

Hi, Currently in 0.10 we can get the information about the consumer groups and respective lag using the kafka-consumer-groups.sh. Is there a way to achieve the same programatically in java? Thanks, Sumit

Re: Kafka Client 0.8.2.2 talks to Kafka Server 0.10.1.1

0.8.x Java clients are supported for both producing and consuming with a 0.10.x broker. If you are seeing data when you poll but it's just in a different format between 0.8 and 0.10 consumers then you might not be using the same (or compatible) serdes libraries to deserialize the messages.

jmxterm workaround for getting metrics; Problem with custom metrics reporter

Hi, Thanks Sachin, the workaround that you proposed to get the metric values from the JMX beans works: You need to specify the type before the client id. However, without this workaround it does not work, so this will probably become an obstacle for the users. The original problem description:

Re: Kafka Client 0.8.2.2 talks to Kafka Server 0.10.1.1

Are you using new java consumer API? It is officially released as part of 0.9 release. 0.8.2.2 java consumer code may not be usable. You have to use old scala consumer API. On Wed, Feb 8, 2017 at 9:35 PM, Jeffrey Zhang wrote: > Any insight about these two versions,

Re: Kafka Client 0.8.2.2 talks to Kafka Server 0.10.1.1

Any insight about these two versions, v0822 vs. v0101, of the Kafka behaviors as described below? Thanks Jeffrey > On Feb 7, 2017, at 7:42 PM, Jeffrey Zhang wrote: > > hi > > I have difficulty to have Kafka Client 0.8.2.2 to consume the messages on a > Kafka Server

FINAL REMINDER: CFP for ApacheCon closes February 11th

Dear Apache Enthusiast, This is your FINAL reminder that the Call for Papers (CFP) for ApacheCon Miami is closing this weekend - February 11th. This is your final opportunity to submit a talk for consideration at this event. This year, we are running several mini conferences in conjunction with

Re: KIP-122: Add a tool to Reset Consumer Group Offsets

Thanks for the feedback! @Onur, @Gwen: Agree. Actually at the first draft I considered to have it inside ´kafka-consumer-groups.sh´, but I decide to propose it as a standalone tool to describe it clearly and focus it on reset functionality. But now that you mentioned, it does make sense to have

Re: Kafka Connect - Unknown magic byte

Below's the rest of my consumer, which includes the serde code. It's worth noting that when I run the following command, it properly outputs the topic to the terminal. sudo kafka-avro-console-consumer --bootstrap-server localhost:9092 \ --from-beginning \

Getting CommitFailedException in 0.10.2.0 due to member id is not valid or unknown

Hi All, I am trying out the 0.10.2.0 rc. We have a source stream of 40 partitions. We start one instance with 4 threads. After that we start second instance with same config on a different machine and then same way third instance. After application reaches steady state we start getting