Re: Is it a bad idea to use periods within a consumer group name? "my-service.topic1_consumer_group"

Not that I know of. We at Flurry have been using periods in our group names for a while now and haven't encountered any issues b/c of that. On Tue, Dec 13, 2016 at 5:13 PM, Jeff Widman wrote: > I vaguely remember reading somewhere that it's a bad idea to use periods >

Consumer/Publisher code is not throwing any exception if my Kafka broker is not running

Hi, I am new to Kafka, could you please let me know why my application is not getting any exception like *refuse to connect so and so host, *if the broker is down/unable to connect *Here is my producer code* private static Map producerConfigs() { Map props = new

Re: How does 'TimeWindows.of().until()' work?

Hi, Thanks for the explanation. This illustration makes it super easy to understand how until works. Perhaps we can update the wiki with this illustration. It is basically the retention time for a past window. I used to think until creates all the future windows for that period and when time

Kafka Errors and Hung Brokers

We are using a 3 node Kafka cluster and are encountering some weird issues. 1) On Each node, when we tail the server.log file under /var/log/kafka we see continuous errors like these pic-partition. (kafka.server.ReplicaFetcherThread) [2016-12-14 02:39:30,747] ERROR

Kafka Errors and Hung Brokers

We are using a 3 node Kafka cluster and are encountering some weird issues. 1) On Each node, when we tail the server.log file under /var/log/kafka we see continuous errors like these pic-partition. (kafka.server.ReplicaFetcherThread) [2016-12-14 02:39:30,747] ERROR

Weird message regarding kafka Cluster

Hello Sir: My name is Chieh-Chun Chang and and we have a problem about our Kafka prod cluster. Kafka client version kafka-clients-0.9.0-kafka-2.0.0.jar Kafka version kafka_2.10-0.9.0-kafka-2.0.0.jar Our kafka broker cluster is experiencing under replicated partitions problems and I found out

Re: Website Update, Part 2

Hi, Since we are breaking down the docs, we can no longer use ctrl-f to find where to find specific things we are looking for... maybe it is time to add a site search bar? I think google has something we can embed. On Tue, Dec 13, 2016 at 6:12 PM, Guozhang Wang wrote: >

Website Update, Part 2

Folks, We are continuing to improve our website, and one of it is to break the single gigantic "documentation" page: https://kafka.apache.org/documentation/ into sub-spaces and sub-pages for better visibility. As the first step of this effort, we will be gradually extract each section of this

Re: lag for a specific partitions on newly added host

The new host has been in place for over a week. Lag is still high on partitions that exist on that new host. Should I attempt another reassign? Thanks -jeremy > On Dec 13, 2016, at 5:43 PM, Apurva Mehta wrote: > > How did you add the host and when did you measure the

Re: lag for a specific partitions on newly added host

How did you add the host and when did you measure the lag? If you used the reassign-partitions script, it will move partitions to the new host, but the data copy will take time. in that period, those partitions will lag. However, once the reassign-partitions script finishes, the partitions on the

Re: kafka commands taking a long time

That is certainly odd. What's the latency when using the kafka console producers and consumers? Is it much faster? If it is, I would just strace the kafka-topics command to see where it is spending the time. On Thu, Dec 8, 2016 at 7:21 AM, Stephen Cresswell < stephen.cressw...@gmail.com> wrote:

Kafka ACL's with SSL Protocol is not working

Hi All, I am trying to enable ACL's in my Kafka cluster with along with SSL Protocol. I tried with each and every parameters but no luck, so I need help to enable the SSL(without Kerberos) and I am attaching all the configuration details in this. Kindly Help me. *I tested SSL without ACL, it

Re: failed to delete kafka topic when building from source

How are you trying to delete the topic? Next time this occurs, can you check whether the process has permissions to perform that operation? On Mon, Dec 12, 2016 at 10:55 PM, Sachin Mittal wrote: > Hi, > I recently built an application from source and I get the following >

Re: How does 'TimeWindows.of().until()' work?

> So a given window (with a '.until()' setting) is triggered for closing by > the presence of a record outside the .until() setting? Yes (and all windows do have a setting for it -- if you do not call it in you code, default is 1 day). However, in Kafka Streams, there is no notion of "closing a

Re: ActiveControllerCount is always will be either 0 or 1 in 3 nodes kafka cluster?

Thanks Sven, I will followup and ensure that the document is tightened up. Apurva On Mon, Dec 12, 2016 at 4:57 AM, Sven Ludwig wrote: > Hi, > > in JMX each Kafka broker has a value 1 or 0 for ActiveControllerCount. As > I understood from this thread, the sum of these values

Re: How does 'TimeWindows.of().until()' work?

So a given window (with a '.until()' setting) is triggered for closing by the presence of a record outside the .until() setting? If the timestamps for records jump about by a value larger than the .until value you could have windows being created / deleted quite a bit then? On Tue, Dec 13, 2016

reasonable KStream app config settings?

My app seems to be continuously rebalancing. If I said it processed data maybe 3 minutes / hour I wouldn't be exaggerating. Surely this isn't normal behavior. My config is: config.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, BROKER_IP); config.put(StreamsConfig.ZOOKEEPER_CONNECT_CONFIG,

Re: How does 'TimeWindows.of().until()' work?

First, windows are only created if there is actual data for a window. So you get windows [0, 50), [25, 75), [50, 100) only if there are record falling into each window (btw: window start-time is inclusive while window end time is exclusive). If you have only 2 record with lets say ts=20 and ts=90

Re: Kafka windowed table not aggregating correctly

Just increase the retention time so the window is not dropped and can accept later arriving data. About your example: retention time specified via until() is a minimum retention time! It can happen, that a window is kept longer. -Matthias On 12/12/16 11:49 PM, Sachin Mittal wrote: > Hi, > Well

Re: checking consumer lag on KStreams app?

There is a workaround: http://stackoverflow.com/questions/41097126/how-to-get-the-group-commit-offset-from-kafka0-10-x I never had time to follow up with this issue. Will put it back on the agenda. Should really get fixed. -Matthias On 12/13/16 2:03 AM, Sachin Mittal wrote: > If this is a bug

Re: "Log end offset should not change while restoring"

StreamsConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG The only one cache parameter. Do you use code auto-completion? It's also in the docs: http://docs.confluent.io/current/streams/developer-guide.html#optional-configuration-parameters -Matthias On 12/12/16 8:22 PM, Jon Yeargers wrote: > What is the

Re: Another odd error

As near as I can see it's rebalancing constantly. I'll up that value and see what happens. On Tue, Dec 13, 2016 at 9:04 AM, Damian Guy wrote: > Hi Jon, > > I haven't had much of a chance to look at the logs in detail too much yet, > but i have noticed that your app seems

Re: Another odd error

Hi Jon, I haven't had much of a chance to look at the logs in detail too much yet, but i have noticed that your app seems to be rebalancing frequently. It seems that it is usually around the 300 second mark, which usually would mean that poll hasn't been called for at least that long. You might

consumer throttling based on rate quotas leads to client errors

Hi - We are using kafka_2.11-0.9.0.1. Using the kafka-configs.sh command, we set the consumer_byte_rate for a specific client.id that was misbehaving. The new setting definitely should have led to some throttling. What we found was that connections from that client.id started failing with the

Re: Another odd error

n/m - I understand the logging issue now. Am generating a new one. Will send shortly. On Tue, Dec 13, 2016 at 4:55 AM, Jon Yeargers wrote: > Yes - saw that one. There were plenty of smaller records available though. > > I sent another log this morning with the level

Re: Another odd error

Yes - saw that one. There were plenty of smaller records available though. I sent another log this morning with the level set to DEBUG. Hopefully you rec'd it. On Tue, Dec 13, 2016 at 3:58 AM, Damian Guy wrote: > HI Jon, > > It looks like you have the logging level for

Re: Another odd error

HI Jon, It looks like you have the logging level for KafkaStreams set to at least WARN. I can only see ERROR level logs being produced from Streams. However, i did notice an issue in the logs (not related to your specific error but you will need to fix anyway): There are lots of messages like:

Re: Another odd error

(am attaching a debug log - note that app terminated with no further messages) topology: kStream -> groupByKey.aggregate(minute) -> foreach \-> groupByKey.aggregate(hour) -> foreach config: Properties config = new Properties();

Re: checking consumer lag on KStreams app?

Is the following PR present https://github.com/apache/kafka/pull/1336? __ Sam Pegler PRODUCTION ENGINEER T. +44(0) 07 562 867 486 3-7 Herbal Hill / London / EC1R 5EJ www.infectiousmedia.com This email and any attachments are confidential and may also be

Re: checking consumer lag on KStreams app?

If this is a bug then it is not fixed because I just build kafka from source and it gave me the reported error. On Tue, Dec 13, 2016 at 3:24 PM, Sam Pegler wrote: > >You can only check the offsets when there is an active member of the > consumer group. > > This

Re: checking consumer lag on KStreams app?

>You can only check the offsets when there is an active member of the consumer group. This was a bug [1] thats been fixed. Thanks to Vincent Dautremont for pointing this out to me a while ago.

Re: checking consumer lag on KStreams app?

Hi Sachin That is correct. You can only check the offsets when there is an active member of the consumer group. In this case that would mean that you have at least one instance of your streams application running. Thanks, Damian On Tue, 13 Dec 2016 at 06:58 Sachin Mittal

Re: rocksdb error(s)

Hi Jon, But i don't see any attached logs? Thanks, Damian On Mon, 12 Dec 2016 at 19:35 Jon Yeargers wrote: > The attached error log shows several of the problems I've run into. After > hitting this list (and typically much less) the app dies. > > On Mon, Dec 12, 2016

Re: How does 'TimeWindows.of().until()' work?

Hi, So is until for future or past? Say I get first record at t = 0 and until is 100 and my window size is 50 advance by 25. I understand it will create windows (0, 50), (25, 75), (50, 100) Now at t = 101 it will drop (0, 50), (25, 75), (50, 100) and create (101, 150), (125, 175), (150, 200)