Streams Processing meetup on Wednesday, February 5, 2020 at LinkedIn, Sunnyvale

*[bcc: (users,dev)@kafka.apache.org ]* Hello, The Streams Infra team invites you to attend the Streams Processing meetup to be held on Wednesday, February 5, 2020. This meetup will focus on Apache Kafka, Apache Samza and related streaming technologies. *Where*: Unify

Streams meetup at LinkedIn Sunnyvale, 6pm, Thursday, October 3, 2019

*[bcc: (users,dev)@kafka.apache.org ]* Hi everyone, The Streams Infra team invites you to attend a Streams Processing meetup on Thursday, October 3, 2019 at LinkedIn's Sunnyvale campus. (This meetup focuses on Apache Kafka, Apache Samza, and related streaming

Streams Meetup at LinkedIn Sunnyvale, 6pm, Wednesday, March 20, 2019

*[bcc: (users,dev)@kafka.apache.org ]* Hi everyone, The Streams Infrastructure team at LinkedIn invites you to attend a Streams Processing meetup on Wednesday, March 20 at LinkedIn’s Sunnyvale campus. (This meetup focuses on Apache Kafka, Apache Samza, and related

Stream processing meetup at LinkedIn (Sunnyvale) on Thursday, February 16 at 6pm

Hi everyone, We would like to invite you to a Stream Processing Meetup at LinkedIn’s Sunnyvale campus on Thursday, February 16 at 6pm. Please RSVP here (*only if you intend to attend in person*): https://www.meetup.com/Stream-Processing-Meetup-LinkedIn/events/237171557/

Re: [ANNOUNCE] New committer: Grant Henke

Hey Grant - congrats! On Thu, Jan 12, 2017 at 10:00 AM, Neha Narkhede wrote: > Congratulations, Grant. Well deserved! > > On Thu, Jan 12, 2017 at 7:51 AM Grant Henke wrote: > > > Thanks everyone! > > > > On Thu, Jan 12, 2017 at 2:58 AM, Damian Guy

Re: [VOTE] Vote for KIP-101 - Leader Epochs

(adding the dev list back - as it seems to have gotten dropped earlier in this thread) On Thu, Jan 5, 2017 at 6:36 PM, Joel Koshy <jjkosh...@gmail.com> wrote: > +1 > > This is a very well-written KIP! > Minor: there is still a mix of terms in the doc that refere

Re: [VOTE] Vote for KIP-101 - Leader Epochs

+1 This is a very well-written KIP! Minor: there is still a mix of terms in the doc that references the earlier LeaderGenerationRequest (which is what I'm assuming what it was called in previous versions of the wiki). Same for the diagrams which I'm guessing are a little harder to make consistent

[ANNOUNCE] New committer: Jiangjie (Becket) Qin

The PMC for Apache Kafka has invited Jiangjie (Becket) Qin to join as a committer and we are pleased to announce that he has accepted! Becket has made significant contributions to Kafka over the last two years. He has been deeply involved in a broad range of KIP discussions and has contributed

Re: Stream processing meetup at LinkedIn (Sunnyvale) on Wednesday, November 2 at 6pm

ns ? > > Cheers, > João Reis > > ________ > From: Joel Koshy <jjkosh...@gmail.com> > Sent: Monday, October 17, 2016 10:25:10 PM > Cc: eyakabo...@linkedin.com > Subject: Stream processing meetup at LinkedIn (Sunnyvale) on Wednesday, > N

Stream processing meetup at LinkedIn (Sunnyvale) on Wednesday, November 2 at 6pm

Hi everyone, We would like to invite you to a Stream Processing Meetup at LinkedIn’s Sunnyvale campus on Wednesday, November 2 at 6pm. Please RSVP here (if you intend to attend in person): http://www.meetup.com/Stream-Processing-Meetup-LinkedIn/events/234454163 We have the following three talks

Stream processing meetup at LinkedIn (Mountain View) on Tuesday, August 23 at 6pm

Hi everyone, We would like to invite you to a Stream Processing Meetup at LinkedIn’s *Mountain View campus on Tuesday, August 23 at 6pm*. Please RSVP here (only if you intend to attend in person): https://www.meetup.com/Stream-Processing-Meetup-LinkedIn/events/232864129 We have three great talks

Re: [kafka-clients] [VOTE] 0.10.0.1 RC2

+1 (binding) Thanks Ismael! On Thu, Aug 4, 2016 at 6:54 AM, Ismael Juma wrote: > Hello Kafka users, developers and client-developers, > > This is the third candidate for the release of Apache Kafka 0.10.0.1. This > is a bug fix release and it includes fixes and improvements

Kafka/Samza meetup at LinkedIn, June 15, 6pm

[bcc: users@kafka.apache.org, d...@kafka.apache.org] Hi everyone, We would like to invite you to our first Stream Processing Meetup at LinkedIn on June 15 at 6pm. Please RSVP here: http://www.meetup.com/Stream-Processing-Meetup-LinkedIn/events/231454378 Going forward (at LinkedIn) we will host

Re: Wiki Karma

You should have access now. On Thu, Feb 18, 2016 at 12:09 PM, Christian Posta wrote: > Can someone add Karma to my user id for contributing to the wiki/docs? > userid is 'ceposta' > > thanks! > > -- > *Christian Posta* > twitter: @christianposta >

Re: Kafka response ordering guarantees

> > Does this mean that when a client is sending more than one in-flight >> request per connection, the server does not guarantee that responses will >> be sent in the same order as requests? > > > No - the server does provide this guarantee - i.e., responses will always > be sent in the same

Re: Kafka response ordering guarantees

> Does this mean that when a client is sending more than one in-flight > request per connection, the server does not guarantee that responses will > be sent in the same order as requests? No - the server does provide this guarantee - i.e., responses will always be sent in the same order as

Re: [kafka-clients] 0.9.0.1 RC1

+1 On Thu, Feb 11, 2016 at 6:55 PM, Jun Rao wrote: > This is the first candidate for release of Apache Kafka 0.9.0.1. This a > bug fix release that fixes 70 issues. > > Release Notes for the 0.9.0.1 release >

Re: Question about offset expiration

Hi Matvey, I have a question about the config value offsets.retention.minutes in kafka > 0.9.0.0. > Is this the timeout for when offsets get compacted in the topic or actually > deleted (as it appears from a cursory reading of > >

Re: trouble upgrading from 0.8.2.1 to 0.9.0.0: invalid message

Hi Dave, This change was introduced in https://issues.apache.org/jira/browse/KAFKA-1755 for compacted topics. > > Interestingly, none of the messages currently going to the topic use > message > compaction (i.e. they all have empty keys), although at some time in the > past > I may have sent a

Re: Kafka APIs version

That is definitely no longer true. A number of requests are at version 1 or higher. I will file a jira for this. On Wed, Jan 20, 2016 at 8:30 AM, tao xiao wrote: > Hi team, > > In the Kafka protocol wiki it states that version 0 is the only supported > version in all APIs.

Re: Create Kafka Topic Programatically

https://issues.apache.org/jira/browse/KAFKA-2945 and related jiras is probably what you are looking for. That is planned for the next release. On Wed, Jan 20, 2016 at 8:41 AM, Tommy Becker wrote: > This works, but it's clumsy and has limitations. Unfortunately, I'm not >

Re: KAFKA-1499 compression.type

Yes you are right - thanks for pointing it out. We will get that fixed. Joel On Fri, Jan 15, 2016 at 10:25 AM, Elias Levy wrote: > Anyone? > > On Thu, Jan 14, 2016 at 8:42 PM, Elias Levy > wrote: > > > The description of the

Re: Does quota requires 0.9.X clients?

I'm pretty sure it should work - you may want to give it a try locally though. We did add a throttle-time field in the responses but that will only be included in responses for requests from 0.9.x clients. 0.8.x requests will just get throttled at the broker and will get an 0.8.x format response

Re: log compaction scaling with ~100m messages

Using log compaction is well-suited for applications that use Kafka directly and need to persist some state associated with its processing. So something like offset management for consumers is a good fit. Another good use-case is for

Re: Dealing with large messages

The best practice I think is to just put large objects in a blob store and have messages embed references to those blobs. Interestingly we ended up having to implement large-message-support at LinkedIn but for various reasons were forced to put messages inline (i.e., against the above

Re: [kafka-clients] [VOTE] 0.8.2.2 Candidate 1

+1 binding On Thu, Sep 3, 2015 at 9:22 AM, Jun Rao wrote: > This is the first candidate for release of Apache Kafka 0.8.2.2. This only > fixes two critical issues (KAFKA-2189 and KAFKA-2308) related to snappy in > 0.8.2.1. > > Release Notes for the 0.8.2.2 release >

Re: New producer in production

It is pretty safe :) and adopting it now will save you the trouble of migrating later. At LinkedIn, we are using the new producer for all the producers that the Kafka team directly own - this means mirror-maker pipelines, REST proxies and some auditing tools. We have not yet migrated other

Re: Offset not committed

committed. Unless I am doing something terribly wrong this is very unreliable On Tue, Jul 14, 2015 at 1:49 PM, Joel Koshy jjkosh...@gmail.com wrote: Actually, how are you committing offsets? Are you using the old (zookeeperconsumerconnector) or new KafkaConsumer? It is true

Re: Fwd: Offset not committed

Can you take a look at the kafka commit rate mbean on your consumer? Also, can you consume the offsets topic while you are committing offsets and see if/what offsets are getting committed? (http://www.slideshare.net/jjkoshy/offset-management-in-kafka/32) Thanks, Joel On Tue, Jul 14, 2015 at

Re: Fwd: Offset not committed

it seem to depend on the number of consumed messages? On Tue, Jul 14, 2015 at 11:36 AM, Joel Koshy jjkosh...@gmail.com wrote: Can you take a look at the kafka commit rate mbean on your consumer? Also, can you consume the offsets topic while you are committing offsets and see if/what offsets

Re: some problem abort isrshrinkrate

I'm assuming you mean you shut down the follower and the leader did not reflect this in isr shrink-rate. I think the reason for this is that the controller would have sent a leaderAndIsr request to the (existing leader) with the shrunk ISR. On the (redundant) leader transition the leader updates

Re: Indication in FetchResponse that fetch size is too small

A somewhat related request came up in another thread and I think it is reasonable to provide this. However, there are already some indicators that you can use: - The consumer iterator throws a MessageSizeTooLargeException if it cannot extract any messages out of the next chunk. - If you are

Re: EOL JDK 1.6 for Kafka

+1 On Wednesday, July 1, 2015, Harsha ka...@harsha.io wrote: Hi, During our SSL Patch KAFKA-1690. Some of the reviewers/users asked for support this config

Re: Is trunk safe for production?

Yes new features are a big part of it and sometimes bug fixes/improvements. Bug fixes are mostly due to being on trunk, but some aren't necessarily introduced on trunk. For e.g., we would like to do a broader roll-out of the new producer, but KAFKA-2121 (adding a request timeout to NetworkClient)

Re: data loss - replicas

: NoCompressionCodec crc: 895349554 Thanks On 06/23/2015 04:52 AM, Joel Koshy wrote: The replicas do not have to decompress/recompress so I don't think that would contribute to this. There may be some corner cases such as: - Multiple unclean leadership elections in sequence - Changing the compression

Re: data loss - replicas

The replicas do not have to decompress/recompress so I don't think that would contribute to this. There may be some corner cases such as: - Multiple unclean leadership elections in sequence - Changing the compression codec for a topic on the fly - different brokers may see this config change at

Re: offset storage as kafka with zookeeper 3.4.6

Is it mandatory to use the zookeeper that comes with kafka for offset storage to be migrated to kafka? If you want to move offsets from zookeeper to Kafka then yes you need to have a phase where all consumers in your group set dual commit to true. If you are starting a fresh consumer group then

Re: Increased replication factor. Replication didn't happen!

This may help: http://kafka.apache.org/documentation.html#basic_ops_increase_replication_factor On Thu, Jun 11, 2015 at 11:20:05AM +0800, Shady Xu wrote: Right now, Kafka topics do not support changing replication factor or partition number after creation. The kafka-reassign-partitions.sh tool

Re: Consumer lag lies - orphaned offsets?

On Fri, Jun 05, 2015 at 12:53:00AM -0400, Otis Gospodnetić wrote: Hi Joel, On Thu, Jun 4, 2015 at 8:52 PM, Joel Koshy jjkosh...@gmail.com wrote: Hi Otis, Yes this is a limitation in the old consumer. i.e., a number of per-topic/partition mbeans remain even on a rebalance. Those need

Re: Consumer lag lies - orphaned offsets?

Hi Otis, Yes this is a limitation in the old consumer. i.e., a number of per-topic/partition mbeans remain even on a rebalance. Those need to be de-registered. So if you stop consuming from some partition after a rebalance that lag mbean currently remain which is why it remains flat. This is a

Re: Kafka Not Commiting Messages

I think you can also set this dynamically via an mbean (kafka.Log4jController) On Thu, May 28, 2015 at 08:26:00PM +, Jiangjie Qin wrote: Actually the name should be log4j.logger.kafka.network.RequestChannel$ It should be there in 0.8.2.1. Can you check it again? From: Charlie Mason

Re: Waiting for replication factor to take effect...?

Do you have state change logs available? You should have a become follower state transition on 9255216 and a leader state transition at around the same time on 925537. The server log should also show 9255216 starting a replica fetcher thread to the leader. Do you have request logging available?

Re: Waiting for replication factor to take effect...?

There are also various mbeans you can poke to check the replica fetcher lag but from your description it appears that the fetcher quit for some reason (since you mentioned the topic is 'pretty much' empty) BTW, the below is only for partition 0 Do you have state change logs available? You

Re: Is fetching from in-sync replicas possible?

Subject: Re: Is fetching from in-sync replicas possible? On May 26, 2015, at 1:44 PM, Joel Koshy jjkosh...@gmail.com wrote: Apologies if this question has been asked before. If I understand things correctly a client can only fetch from the leader of a partition, not from an (in-sync) replica

Re: Is fetching from in-sync replicas possible?

back. Thanks - I was wondering if you had done some simple ping/traceroute tests. It's just that with replicas in different zones the end-to-end latency from producer to consumer will be correspondingly higher. Does your zookeeper setup span zones as well? On Tue, May 26, 2015 at 10:44 PM, Joel

Re: Is fetching from in-sync replicas possible?

Apologies if this question has been asked before. If I understand things correctly a client can only fetch from the leader of a partition, not from an (in-sync) replica. I have a use case where it would be very beneficial if it were possible to fetch from a replica instead of just the leader,

Re: Mirrormaker stops consuming

between all producer and consumer threads..? Thanks, Raja. On Fri, May 22, 2015 at 12:12 PM, Joel Koshy jjkosh...@gmail.com wrote: The threaddump suggests that one of the producers (mirrormaker-producer-6) is blocked on write for some reason. So the data-channel for that producer

Re: Replica manager exception in broker

When you say keeps getting below exception I'm assuming that the error offset (206845418) keeps changing - right? We saw a similar issue in the past and it turned out to be due to a NIC issue - i.e., it negotiated at a low speed. So the replica fetcher couldn't keep up with the leader. i.e., while

Re: Mirrormaker stops consuming

The threaddump suggests that one of the producers (mirrormaker-producer-6) is blocked on write for some reason. So the data-channel for that producer (which sits between the consumers and the producer) is full which blocks the consumers from progressing. This appears to be in your (custom)

Re: Replica manager exception in broker

didn't grow for several hours On Sat, May 23, 2015 at 12:40 AM, Joel Koshy jjkosh...@gmail.com wrote: When you say keeps getting below exception I'm assuming that the error offset (206845418) keeps changing - right? We saw a similar issue in the past and it turned out to be due to a NIC

Re: New Producer API - batched sync mode support

This sounds like flush: https://cwiki.apache.org/confluence/display/KAFKA/KIP-8+-+Add+a+flush+method+to+the+producer+API which was recently implemented in trunk. Joel On Mon, Apr 27, 2015 at 08:19:40PM +, Roshan Naik wrote: Been evaluating the perf of old and new Produce APIs for reliable

Re: New Producer API - batched sync mode support

On 4/27/15 1:36 PM, Joel Koshy jjkosh...@gmail.com wrote: This sounds like flush: https://cwiki.apache.org/confluence/display/KAFKA/KIP-8+-+Add+a+flush+meth od+to+the+producer+API which was recently implemented in trunk. Joel On Mon, Apr 27, 2015 at 08:19:40PM +, Roshan Naik

Re: New Producer API - batched sync mode support

Fine grained tracking of status of individual events is quite painful in contrast to simply blocking on every batch. Old style Batched-sync mode has great advantages in terms of simplicity and performance. I may be missing something, but I'm not so convinced that it is that painful/very

Re: New Offset Management API Question

1) Does Offset Commit/Fetch API works with Simple Consumer ? Yes - in 0.8.2.1. There is an example given at https://cwiki.apache.org/confluence/display/KAFKA/Committing+and+fetching+consumer+offsets+in+Kafka 2) With MM, can you selectively MM offset topic. Use Case: Let's suppose you want

Re: Log cleaner patch (KAFKA-1641) on 0.8.2.1

+1 - if you have a way to reproduce that would be ideal. We don't know the root cause of this yet. Our guess is a corner case around shutdowns, but not sure. On Fri, Mar 13, 2015 at 03:13:45PM -0700, Jun Rao wrote: Is there a way that you can reproduce this easily? Thanks, Jun On Fri,

Re: High Replica Max Lag

, others vary between 5 to 14). Thanks Zakee On Mar 13, 2015, at 3:25 PM, Joel Koshy jjkosh...@gmail.com wrote: I think what people have observed in the past is that increasing num-replica-fetcher-threads has diminishing returns fairly quickly. You may want to instead increase

Re: High Replica Max Lag

I think what people have observed in the past is that increasing num-replica-fetcher-threads has diminishing returns fairly quickly. You may want to instead increase the number of partitions in the topic you are producing to. (How many do you have right now?) On Fri, Mar 13, 2015 at 02:48:17PM

Re: Trying to get kafka data to Hadoop

I think the camus mailing list would be more suitable for this question. Thanks, Joel On Wed, Mar 04, 2015 at 11:00:51AM -0500, max square wrote: Hi all, I have browsed through different conversations around Camus, and bring this as a kinda Kafka question. I know is not the most orthodox,

Re: moving replications

I think what you may be looking for is being discussed here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-6+-+New+reassignment+partition+logic+for+rebalancing On Wed, Mar 04, 2015 at 12:34:30PM +0530, sunil kalva wrote: Is there any way to automate On Mar 3, 2015 11:57 AM, sunil kalva

Re: high level consumer rollback

This is not possible with the current high-level consumer without a restart, but the new consumer (under development) does have support for this. On Wed, Mar 04, 2015 at 03:04:57PM -0500, Luiz Geovani Vier wrote: Hello, I'm using the high level consumer with auto-commit disabled and a single

Re: Why The Division Between Scala And Java

We will eventually only have Java clients. For your specific question: javaapi.SimpleConsumer and consumer.SimpleConsumer - there are some arguments that contain scala-specific constructs. E.g., scala maps which cannot be created in Java. This is why we expose a javaapi variant which takes Java

Re: Simple Consumer and offsets

. -Suren On Thursday, February 19, 2015 10:25 AM, Joel Koshy jjkosh...@gmail.com wrote: Not sure what you mean by using the SimpleConsumer on failure recovery. Can you elaborate on this? On Thu, Feb 19, 2015 at 03:04:47PM +, Suren wrote: Haven't used

Re: Commit offset with SimpleConsumer in 0.8.2

Can you add yourself as a watcher on KAFKA-1729? I will update that when I fix the example on the wiki. On Sun, Feb 22, 2015 at 10:16:44PM +0100, Jochen Mader wrote: I have a hard time figuring out how to do a commit using API 0.8.2 on JDK 8. I tried using the examples from 0.8.1.1. First

Re: ability to delete inactive queues?

We do support delete topic. However, this is a client(admin) operation that is done via zookeeper. It would be useful to do this automatically on the broker-side. Can you file a jira for this? It is not very straightforward to implement this since you would want to check across all partitions

Re: Simple Consumer and offsets

the SimpleConsumer on failure recovery to set the offsets. Is that the recommended approach for this use case? Thanks. -Suren On Thursday, February 19, 2015 9:40 AM, Joel Koshy jjkosh...@gmail.com wrote: Are you using it from Java or Scala? i.e., are you using

Re: Simple Consumer and offsets

a SimpleConsumer just to reset offsets on restart was a viable option, while continuing to use the High Level Consumer for our normal operations. Not sure if there is a better way that is compatible across 0.8.1 and 0.8.2. -Suren On Thursday, February 19, 2015 10:25 AM, Joel Koshy

Re: Consuming a snapshot from log compacted topic

actually has this information internally (we return it with the fetch request) but doesn't expose it. -Jay On Wed, Feb 18, 2015 at 4:51 PM, Joel Koshy jjkosh...@gmail.com wrote: 2. Make the log end offset available more easily in the consumer

Re: New Consumer Offset management in 0.8.2

Yes it is supported in 0.8.2-beta. It is documented on the site - you will need to set offsets.storage to kafka. On Thu, Feb 19, 2015 at 03:57:31PM -0500, Matthew Butt wrote: I'm having a hard time figuring out if the new Kafka-based offset management in the high-level Scala Consumer is

Re: Simple Consumer and offsets

:22 PM, Joel Koshy jjkosh...@gmail.com wrote: Hi Chris, In 0.8.2, the simple consumer Java API supports committing/fetching offsets that are stored in ZooKeeper. You don't need to issue any ConsumerMetadataRequest for this. Unfortunately, the API currently does not support fetching

Re: Simple Consumer and offsets

Yeah that is a good point - will do the update as part of the doc changes in KAFKA-1729 On Thu, Feb 19, 2015 at 09:26:30PM -0500, Evan Huus wrote: On Thu, Feb 19, 2015 at 8:43 PM, Joel Koshy jjkosh...@gmail.com wrote: If you are using v0 of OffsetCommit/FetchRequest then you can issue

Re: Simple Consumer and offsets

/All, The SimpleConsumer constructor requires a specific host and port. Can this be any broker? If it needs to be a specific broker, for 0.8.2, should this be the offset coordinator? For 0.8.1, does it matter? -Suren On Thursday, February 19, 2015 10:43 AM, Joel Koshy jjkosh

Re: Consuming a snapshot from log compacted topic

at 18:47, Joel Koshy jjkosh...@gmail.com wrote: If I consumed up to the log end offset and log compaction happens in between, I would have missed some messages. Compaction actually only runs on the rolled over segments (not the active - i.e., latest segment). The log-end-offset

Re: Consuming a snapshot from log compacted topic

You are also correct and perceptive to notice that if you check the end of the log then begin consuming and read up to that point compaction may have already kicked in (if the reading takes a while) and hence you might have an incomplete snapshot. Isn't it sufficient to just repeat the check

Re: Consuming a snapshot from log compacted topic

the max offset values, which is still long enough for more messages to arrive. Got it - thanks for clarifying. On 18 February 2015 at 23:16, Joel Koshy jjkosh...@gmail.com wrote: You are also correct and perceptive to notice that if you check the end of the log then begin consuming

Re: Simple Consumer and offsets

Hi Chris, In 0.8.2, the simple consumer Java API supports committing/fetching offsets that are stored in ZooKeeper. You don't need to issue any ConsumerMetadataRequest for this. Unfortunately, the API currently does not support fetching offsets that are stored in Kafka. Thanks, Joel On Mon,

Re: consumer lag metric

There are FetcherLagMetrics that you can take a look at. However, it is probably easiest to just monitor MaxLag as that reports the maximum of all the lag metrics. On Fri, Feb 13, 2015 at 05:03:28PM +0800, tao xiao wrote: Hi team, Is there a metric that shows the consumer lag of a particular

Re: offset migration from kafka to zookeeper

That is weird. Are you by any chance running an older version of the offset checker? Is this straightforward to reproduce? On Fri, Feb 13, 2015 at 09:57:31AM +0800, tao xiao wrote: Joel, No, the metric was not increasing. It was 0 all the time. On Fri, Feb 13, 2015 at 12:18 AM, Joel Koshy

Re: Java APIs - which one to use?

You should use the producer under o.a.k.c. The new consumer implementation is not available in 0.8.2 (although the APIs are there) so you would need to use the kafka.javaapi classes for the consumer. We plan to deprecate kafka.javaapi eventually. Thanks, Joel On Wed, Feb 11, 2015 at 11:42:04AM

Re: question about new consumer offset management in 0.8.2

that can be improved. On Thu, Feb 5, 2015 at 2:21 PM, Joel Koshy jjkosh...@gmail.com wrote: This is documented in the official docs: http://kafka.apache.org/documentation.html#distributionimpl On Thu, Feb 05, 2015 at 01:23:01PM -0500, Jason Rosenberg wrote: What

Re: question about new consumer offset management in 0.8.2

On Thu, Feb 05, 2015 at 11:57:15PM -0800, Joel Koshy wrote: On Fri, Feb 06, 2015 at 12:43:37AM -0500, Jason Rosenberg wrote: I'm not sure what you mean by 'default' behavior 'only if' offset.storage is kafka. Does that mean the 'default' behavior is 'false' if offset.storage is 'zookeeper

Re: question about new consumer offset management in 0.8.2

This is documented in the official docs: http://kafka.apache.org/documentation.html#distributionimpl On Thu, Feb 05, 2015 at 01:23:01PM -0500, Jason Rosenberg wrote: What are the defaults for those settings (I assume it will be to continue using only zookeeper by default)? Also, if I have a

Re: question about new consumer offset management in 0.8.2

PM, Joel Koshy jjkosh...@gmail.com wrote: This is documented in the official docs: http://kafka.apache.org/documentation.html#distributionimpl On Thu, Feb 05, 2015 at 01:23:01PM -0500, Jason Rosenberg wrote: What are the defaults for those settings (I assume it will be to continue

Re: Issue with topic deletion

) there won't be any consumer running for the deleted topic. Sumit -Harsha On Tue, Feb 3, 2015, at 06:17 PM, Joel Koshy wrote: Thanks for the logs - will take a look tomorrow unless someone else gets a chance

Re: How to delete defunct topics

There are mbeans (http://kafka.apache.org/documentation.html#monitoring) that you can poke for incoming message rate - if you look at those over a period of time you can figure out which of those are likely to be defunct and then delete those topics. On Thu, Feb 05, 2015 at 02:38:27PM -0800,

Re: How to fetch old messages from kafka

We can reset the offset and get first 10 messages, but since we need to back in reverse sequence, suppose user has consumed messages upto 100 offset , currently there are only last 10 messages are visible, from 100 -90, now I want to retrieve messages from 80 to 90, how can we do that? I'm

Re: Get Latest Offset for Specific Topic for All Partition

https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowdoIaccuratelygetoffsetsofmessagesforacertaintimestampusingOffsetRequest? However, you will need to issue a TopicMetadataRequest first to discover the leaders for all the partitions and then issue the offset request. On Thu, Feb 05,

Re: generics type for Producer and Consumer do not need to match?

There has to be an implicit contract between the producer and consumer. The K, V pairs don't _need_ to match but generally _should_. If producer sends with PK, PV the consumer may receive CK, CV as long as it knows how to convert those raw bytes to CK, CV. In the example if CK == byte[] and CV ==

Re: Issue with topic deletion

Thanks for the logs - will take a look tomorrow unless someone else gets a chance to get to it today. Joel On Tue, Feb 03, 2015 at 04:11:57PM -0800, Sumit Rangwala wrote: On Tue, Feb 3, 2015 at 3:37 PM, Joel Koshy jjkosh...@gmail.com wrote: Hey Sumit, I thought you would be providing

Re: kafka-web-console goes down regularly

Can you contact the maintainer directly? http://github.com/claudemamo/kafka-web-console/issues On Tue, Feb 03, 2015 at 12:09:46PM -0800, Sa Li wrote: Hi, All I am currently using kafka-web-console to monitor the kafka system, it get down regularly, so I have to restart it every few hours

Re: Turning on cleanup.policy=compact for a topic = not starting cleanup ?

. Thanks in advance, Thunder -Original Message- From: Joel Koshy [mailto:jjkosh...@gmail.com] Sent: Tuesday, February 03, 2015 3:07 PM To: users@kafka.apache.org Subject: Re: Turning on cleanup.policy=compact for a topic = not starting cleanup ? - Can you check the log cleaner

Re: Can't create a topic; can't delete it either

If you can tell me where the find the logs I can check. I haven't restarted my brokers since the issue. This will be specified in the log4j properties that you are using. On Wed, Jan 28, 2015 at 12:01:01PM -0800, Sumit Rangwala wrote: On Tue, Jan 27, 2015 at 10:54 PM, Joel Koshy jjkosh

Re: Missing Per-Topic BrokerTopicMetrics in v0.8.2.0

Hi Jason - can you describe how you verify that the metrics are not coming through to the metrics registry? Looking at the metrics code it seems that the mbeans are only registered by the yammer jmx reporter only after being added to the metrics registry. Thanks, Joel On Tue, Jan 27, 2015 at

Re: Can't create a topic; can't delete it either

Which version of the broker are you using? On Mon, Jan 26, 2015 at 10:27:14PM -0800, Sumit Rangwala wrote: While running kafka in production I found an issue where a topic wasn't getting created even with auto topic enabled. I then went ahead and created the topic manually (from the command

Re: Missing Per-Topic BrokerTopicMetrics in v0.8.2.0

Is it actually getting double-counted? I tried reproducing this locally but the BrokerTopicMetrics.Count lines up with the sum of the PerTopic.Counts for various metrics. On Tue, Jan 27, 2015 at 03:29:37AM -0500, Jason Rosenberg wrote: Ok, It looks like the yammer MetricName is not being

Re: Can't create a topic; can't delete it either

the default.replication.factor parameter. Gwen On Tue, Jan 27, 2015 at 12:29 AM, Joel Koshy jjkosh...@gmail.com wrote: Which version of the broker are you using? On Mon, Jan 26, 2015 at 10:27:14PM -0800, Sumit Rangwala wrote: While running kafka in production I found an issue where a topic wasn't

Re: Ques regarding topic partition offset

This can happen as a result of unclean leader elections - there are mbeans on the controller that give the unclean leader election rate - or you can check the controller logs to determine if this happened. On Tue, Jan 27, 2015 at 09:54:38PM -0800, Liju John wrote: Hi , I have query regarding

Re: unable to delete topic with 0.8.2 rc2

Hey Jason, Is it an option for you to do the following: - Bounce in a config change to the brokers to turn off auto-create - (Batch)-delete the topic(s) - Wait long enough for consumers to rebalance (after which they will no longer consume the topic(s)) - Bounce in a config change to the

Re: unable to delete topic with 0.8.2 rc2

Hmm.. that's right. completely forgot about that. On Mon, Jan 26, 2015 at 01:49:33PM -0800, Jun Rao wrote: Joel, That's probably because console consumer always uses wildcard for consumption. Thanks, Jun On Mon, Jan 26, 2015 at 1:44 PM, Joel Koshy jjkosh...@gmail.com wrote: I

Re: Leadership rebalance causing drop of incoming messages

at high rate? On Thu, Jan 15, 2015 at 5:51 PM, Joel Koshy jjkosh...@gmail.com wrote: Not sure what could be going on.. What version of the client and the broker are you on? Can you verify from the state change logs the time it took for leadership to move to the preferred leader

Re: Leadership rebalance causing drop of incoming messages

Is leadership rebalance a safe operation? Yes - we use it routinely. For any partition, there should only be a brief (order of seconds) period of rejected messages as leaders move. When that happens the client should refresh metadata and discover the new leader. Are you using the Java producer?

  1   2   3   >