Re: Failure to reset consumer offsets for specific topics

For beats (the topic where timestamps are OK) the producers are Metricbeat / Logstash which I assume use a relatively recent producer version. For the topics with missing timestamps the producers are based on librdkafka though I'm not sure which version exactly - I wouldn't be surprised if it's pre

Re: [DISCUSS] KIP-213 Support non-key joining in KTable

Looks like Jan technically made his KIP wiki page first so I'll just change my KIP number. On Wed, Oct 25, 2017 at 4:59 PM, Matthias J. Sax wrote: > Thanks a lot for the KIP. Can we please move the discussion to the dev > list? > > Thus, after fixing the KIP collision, just start a new DISCUSS t

Re: Failure to reset consumer offsets for specific topics

Which driver is used yo produce these messages ? On Thu, Oct 26, 2017 at 8:14 AM, Dan Markhasin wrote: > Furthermore, when looking at messages produced to the data1_log topic with > print.timestamp=true they all have CreateTime:-1 whereas messages produced > to the beats topic have valid timesta

Re: Failure to reset consumer offsets for specific topics

Furthermore, when looking at messages produced to the data1_log topic with print.timestamp=true they all have CreateTime:-1 whereas messages produced to the beats topic have valid timestamps. The producers that are sending to data1_log are older than the producers that are sending to beats - if thi

Re: Failure to reset consumer offsets for specific topics

After a bit more checking it seems that Kafka isn't writing timestamps at all in the .timeindex file for the topics where offset rewind is not working. The following output is from * a different 0.11.0.0 * which also has a topic called data1_log (this cluster has not experienced any issues lately)

Re: Kafka Streams 0.11.0.1 - Task Assignment Stickiness Behavior

> On Oct 24, 2017, at 8:53 PM, Matthias J. Sax wrote: > > Might be worth a try with 1.0.0 RC3 -- even if I doubt that much changes. > > Can you provide debug logs for your Kafka streams applications as well > as brokers? This would help to dig into this. > I searched, but it wasn’t clear to m

Re: Failure to reset consumer offsets for specific topics

Any idea why I started receiving mails from this list as of 2:43 am today? I didn't make any changes or subscribe to anything. I even clicked unsubscribe earlier today and am still receiving mails. Maybe there's a misconfiguration in your email list? Kelly On Wed, Oct 25, 2017 at 1:44 PM, Hans Je

Re: Failure to reset consumer offsets for specific topics

Which driver is used yo produce these messages ? On Oct 26, 2017 07:11, "Dan Markhasin" wrote: > No, that flag doesn't affect which offsets are returned, only executes the > action (and resets the consumer to latest offset when used, regardless of > datetime value I provide). > > On 25 October 2

Re: Failure to reset consumer offsets for specific topics

No, that flag doesn't affect which offsets are returned, only executes the action (and resets the consumer to latest offset when used, regardless of datetime value I provide). On 25 October 2017 at 23:44, Hans Jespersen wrote: > I think you are just missing the —execute flag. > > -hans > > > On

Re: Consumer poll returning 0 results

Are you producing any records after you start the consumer? By default, Kafka consumer starts with auto.offset.reset == latest ( https://kafka.apache.org/documentation/#newconsumerconfigs), which means that if the consumer doesn't find a previous offset for its consumer group (e.g. the first time

Re: Consumer poll returning 0 results

Can you provide a bit more information ? Release of Kafka Java / Scala version Thanks On Wed, Oct 25, 2017 at 6:40 PM, Susheel Kumar wrote: > Hello Kafka Users, > > I am trying to run below sample code mentioned in Kafka documentation under > Automatic Offset Committing for a topic with 1 part

Consumer poll returning 0 results

Hello Kafka Users, I am trying to run below sample code mentioned in Kafka documentation under Automatic Offset Committing for a topic with 1 partition (tried with 3 and more partition as well). Create command as follows bin/kafka-topics.sh --create --zookeeper :2181 --replication-factor 3 --par

Re: [VOTE] 1.0.0 RC3

Hello Vahid, As we have discussed on the PR of KAFKA-6075, Windows is not a supported platform for the Kafka broker at this point. As for KAFKA-6100, I'll try to fix it by upgrading RocksDB to 5.7.3 in the new RC (see below). Folks, We just found another blocker issue on the exactly-once featu

Re: [DISCUSS] KIP-213 Support non-key joining in KTable

Thanks a lot for the KIP. Can we please move the discussion to the dev list? Thus, after fixing the KIP collision, just start a new DISCUSS thread. Thx. -Matthias On 10/25/17 4:20 PM, Ted Yu wrote: > Have you seen the email a moment ago from Onur which uses the same KIP > number ? > > Looks l

Re: [DISCUSS] KIP-213 Support non-key joining in KTable

Have you seen the email a moment ago from Onur which uses the same KIP number ? Looks like there was race condition in modifying wiki. Please consider bumping the KIP number. Thanks On Wed, Oct 25, 2017 at 4:14 PM, Jan Filipiak wrote: > Hello Kafka-users, > > I want to continue with the devel

[DISCUSS] KIP-213 Support non-key joining in KTable

Hello Kafka-users, I want to continue with the development of KAFKA-3705, which allows the Streams DSL to perform KTableKTable-Joins when the KTables have a one-to-many relationship. To make sure we cover the requirements of as many users as possible and have a good solution afterwards I invit

Re: Failure to reset consumer offsets for specific topics

Clarification: my most recent reply was w.r.t. the strange situation Dan described, not the offset resetting. On Wed, Oct 25, 2017 at 1:24 PM, Ted Yu wrote: > I wonder if you have hit KAFKA-5600. > > Is it possible that you try out 0.11.0.1 ? > > Thanks > > On Wed, Oct 25, 2017 at 1:15 PM, Dan M

Re: Failure to reset consumer offsets for specific topics

I think you are just missing the —execute flag. -hans > On Oct 25, 2017, at 1:24 PM, Ted Yu wrote: > > I wonder if you have hit KAFKA-5600. > > Is it possible that you try out 0.11.0.1 ? > > Thanks > >> On Wed, Oct 25, 2017 at 1:15 PM, Dan Markhasin wrote: >> >> I am using 0.11.0.0. >> >>

Re: [VOTE] 1.0.0 RC3

Hi Guozhang, +1 for Ubuntu and Mac: successfully built jars and tested quickstarts with this RC (using Java 8). -1 for Windows: because of KAFKA-6075 and KAFKA-6100. To me these two issues (which have simple workarounds) sound like "blockers" - unless Kafka does not officially support Windows.

Re: Failure to reset consumer offsets for specific topics

I wonder if you have hit KAFKA-5600. Is it possible that you try out 0.11.0.1 ? Thanks On Wed, Oct 25, 2017 at 1:15 PM, Dan Markhasin wrote: > I am using 0.11.0.0. > > There is no difference configuration-wise - both have 10 partitions and 2 > replicas. There are no errors in the logs, but loo

Re: Failure to reset consumer offsets for specific topics

I am using 0.11.0.0. There is no difference configuration-wise - both have 10 partitions and 2 replicas. There are no errors in the logs, but looking in the data folder it seems like Kafka is not updating the timeindex file for data1_log - notice how the timeindex file for the current log segment

Re: Failure to reset consumer offsets for specific topics

Do you mind providing a bit more information ? Release of Kafka you use Any difference between data1_log and the other, normal topic ? Probably check the broker log where data1_log is hosted - see if there is some clue. Thanks On Wed, Oct 25, 2017 at 12:11 PM, Dan Markhasin wrote: > I'm tryi

Re: "Persistence strategy" clarification

Ok, it has sense. So I have to think that the Kafka write strategy (writing directly to the disk if I understood) is “overwritten” by the Linux page-caching strategy? Anyway this invalid everything because it store data on the RAM before using disk space... > On 25 Oct 2017, at 20:51, Fabio Ya

Failure to reset consumer offsets for specific topics

I'm trying to use the kafka-consumer-groups.sh tool in order to rewind a consumer group's offset, however it seems to be returning the latest offset regarding of the requested offset. You can see in the below example that two consecutive commands to reset the offset to a specific point in time ret

Re: "Persistence strategy" clarification

Hi Andrea, Although I'm not experienced in Kafka I could find reference in the docs to explain the behavior you experienced in the graph analysis. Following an excerpt from the official documentation: Understanding Linux OS Flush Behavior In Li

"Persistence strategy" clarification

Hi, I’m looking at Kafka documentation, in particular at Persistence section: https://kafka.apache.org/documentation/#persistence If I understood it says that Kafka writes on disks data as they arrive instead of use RAM. It sounds really strange to me (writes on disks are not heavy operations?)

Re: Log Compaction Not Picking up Topic [solved]

Hello Xin, hello Jan, worked perfectly. I did a build of an image based on 0.11.0.1 and applied the missing patch, cleaning went through and resulted in the expected size. Thanks a lot for the quick help, Elmar On 10/25/2017 1:03 PM, Xin Li wrote: Hey Elmar, The only thing you need to do i

Re: Log Compaction Not Picking up Topic

Hey Elmar, The only thing you need to do is upgrade, Kafka track cleaned offset using cleaner-offset-checkpoint file. Best, Xin Xin Li Data EngineeringXin.Li@ trivago.com www.trivago.com

Re: Log Compaction Not Picking up Topic

Hey, Because of the overflow the calculation for dirty ratios is minus, and I guess upgrade is the one time for good fix. And we running that for quite a while, so far so good. Best, Xin Xin Li Data EngineeringXin.Li@ trivago.com <

Re: Log Compaction Not Picking up Topic

Hi, thanks, I'll give it a try, we run on Kubernetes so it's not a big issue to replicate the whole env including data. One question I'd have left: - How can I force a re-compaction over the whole topic? Because I guess the Log Cleaner market everything so far as not able to clean, how wil

Re: Log Compaction Not Picking up Topic

Hi, unfortunatly there is nothing trivial you could do here. Without upgrading your kafkas you can only bounce the partition back and forth between brokers so they compact while its still small. With upgrading you could also just cherrypick this very commit or put a logstatement to verify.

Re: Log Compaction Not Picking up Topic

Hi, On 10/25/2017 12:15 PM, Xin Li wrote: > I think that is a bug, and should be fixed in this task https://issues.apache.org/jira/browse/KAFKA-6030. > We experience that in our kafka cluster, we just check out the 11.0.2 version, build it ourselves. thanks for the hint, as it looks like a c

Re: Log Compaction Not Picking up Topic

Hey, I think that is a bug, and should be fixed in this task https://issues.apache.org/jira/browse/KAFKA-6030. We experience that in our kafka cluster, we just check out the 11.0.2 version, build it ourselves. Best, Xin Xin Li Data EngineeringXin.Li@

Re: Log Compaction Not Picking up Topic

On 10/25/2017 12:03 PM, Manikumar wrote: any errors in log cleaner logs? Not as far as I can see root@kafka-1:/opt/kafka/logs# cat log-cleaner.log* | grep -i error (empty) However, I've seen that it actually did cleaning of the whole topic (excerpts below), but it didn't seem to find anythin

Re: Log Compaction Not Picking up Topic

any errors in log cleaner logs? On Wed, Oct 25, 2017 at 3:12 PM, Elmar Weber wrote: > Hello, > > I'm having trouble getting Kafka to compact a topic. It's over 300GB and > has enough segments to warrant cleaning. It should only be about 40 GB > (there is a copy in a db that is unique on the key)

Log Compaction Not Picking up Topic

Hello, I'm having trouble getting Kafka to compact a topic. It's over 300GB and has enough segments to warrant cleaning. It should only be about 40 GB (there is a copy in a db that is unique on the key). Below are the configs we have (default broker) and topic override. Is there something I