Re: User interface for kafka: Management/administration

Definitely more focused on the consuming side vs management, but has some management tooling and has a binary distribution: https://github.com/sourcelaborg/kafka-webview On Thu, Jan 23, 2020 at 1:10 PM Sunil CHAUDHARI wrote: > Hi all, > Please help me to get some user interface for management

Re: help for regarding my question

true On Thu, Nov 14, 2019 at 2:03 PM prashanth sri wrote: > i am new to kafka , can any one help me , regarding my problem. >

Re: Any simple UI Tool for Kafka

Pretty sure Kafka-WebView checks most of those boxes :) https://github.com/SourceLabOrg/kafka-webview On Sat, Aug 24, 2019 at 12:58 PM Darius Cooper wrote: > I'm looking for recommendations for a simple UI -based tool that will help > > * viewing lists of Kafka topics, > * viewing Consumer

Re: kafka server shutdown automatically

Looks like someone/something on your system sent it a SIGHUP signal: [2019-05-09 12:54:56,295] INFO *Terminating process due to signal SIGHUP * > (org.apache.kafka.common.utils.LoggingSignalHandler) > > On Fri, May 10, 2019 at 5:43 PM wrote: > Hi guys, > my kafka server dead without any error

Re: Kafka tests on a remote cluster

Without looking into how the integration tests work my best guess is within the context they were written to run in, it doesn't make sense to run them against a remote cluster. The "internal" cluster is running the same code, so why require having to coordinate with an external dependency? For

Re: Kafka UI

Shameless plug: https://github.com/sourcelaborg/kafka-webview The project is definitely more geared around viewing data within topics but you can view details about your clusters as well. You can view things such as node membership, configuration of the nodes, topics defined on the cluster,

Re: [DISCUSS] KIP-369 Alternative Partitioner to Support "Always Round-Robin" Selection

Neat, this would be super helpful! I submitted this ages ago: https://issues.apache.org/jira/browse/KAFKA- On Fri, Aug 31, 2018 at 5:04 AM, Satish Duggana wrote: > +including both dev and user mailing lists. > > Hi, > Thanks for the KIP. > > "* For us, the message keys represent some

Re: Mirrormaker producing to only one partition in a topic

Hey Ryan, I ran into a similar issue and it was how the RoundRobinAssignor/Partitioner was hashing the keys in my messages. You may want to look at how thats implemented and see if its causing all of your messages to end up in the same partition. For what its worth, this ticket has the

Re: Using Kafka CLI without specifying the URLs every single time?

We also have created simple wrapper scripts for common operations. On Sat, Apr 21, 2018 at 2:20 AM, Peter Bukowinski wrote: > One solution is to build wrapper scripts around the standard kafka > scripts. You’d put your relevant cluster parameters (brokers, zookeepers) > in a

Re: What's the use of timestamp in ProducerRecord?

A common use case is for calculating a "processing lag time" comparing the record's timestamp (when it was published/persisted) against the current time that the consumer 'consumed' it. In theory, the delta is 'how far behind' the consumer is, tho you can come up with a variety of reasons why

Re: Figuring out lag within Java consumer application

So I have the same use case as the original poster and had the same issue with the older 0.10.x clients and not being able to determine the tail offsets even tho the fetch response contains the HW mark. >From what I could understand by tracing through the 0.11.0 consumer code, it makes additional

Re: Consumer service that supports retry with exponential backoff

Pretty much the implementation that we're using, exception using partitions within a single topic because we have a low number of failures passing through. Granted the first partition will take the brunt of the load vs the higher order partitions. On Tue, Oct 10, 2017 at 7:30 AM, Steven

Re: creating a controlled lag in for a consumer group

Hey Ido. I haven't tried to do something like this, but we've worked out some future plans to do something similar, so I have a bit of interest in what you're saying. I feel like there are some details left out from your post. Are your consumers able to keep up w/ the throughput without the

Re: Help please: Topics deleted, but Kafka continues to try and sync them with Zookeeper

I could be totally wrong, but I seem to recall that delete wasn't fully implemented in 0.8.x? On Fri, Jul 21, 2017 at 10:10 AM, Carl Haferd wrote: > Chris, > > You could first check to make sure that delete.topic.enable is true and try > deleting again if not. If

Is this a decent use case for Kafka Streams?

Hey! I was hoping I could get some input from people more experienced with Kafka Streams to determine if they'd be a good use case/solution for me. I have multi-tenant clients submitting data to a Kafka topic that they want ETL'd to a third party service. I'd like to batch and group these by

Re: Taking a long time to roll a new log segment (~1 min)

l times look > at the dtrace log to see if it was that call? > > -Dave > > -Original Message- > From: Stephen Powis [mailto:spo...@salesforce.com] > Sent: Friday, January 13, 2017 9:25 AM > To: users@kafka.apache.org > Subject: Re: Taking a long time to roll a new log

Re: Taking a long time to roll a new log segment (~1 min)

low (below 0.01%) with no recorded spikes. On Fri, Jan 13, 2017 at 6:57 AM, Ismael Juma <ism...@juma.me.uk> wrote: > Hi Stephen, > > I think it's worth capturing this in a JIRA so that it doesn't get lost. > > Ismael > > On Thu, Jan 12, 2017 at 7:22 PM, Stephen Powis <spo

Re: Taking a long time to roll a new log segment (~1 min)

Running Centos 6.7 3.10.95-1.el6.elrepo.x86_64. 4 SATA disks in RAID10 with ZFS On Thu, Jan 12, 2017 at 2:27 PM, Tauzell, Dave <dave.tauz...@surescripts.com > wrote: > You have a local filesystem? Linux? > > -Dave > > -Original Message- > From: S

Re: Taking a long time to roll a new log segment (~1 min)

I've further narrowed it down to this particular line: https://github.com/apache/kafka/blob/0.10.0/core/src/main/scala/kafka/log/OffsetIndex.scala#L294 But I'm still at a loss to why this would be slow sometimes and not others. On Thu, Jan 12, 2017 at 10:56 AM, Stephen Powis <

Re: Taking a long time to roll a new log segment (~1 min)

; 2017-01-12T07:22 = 32.409 > 2017-01-12T07:23 = 85.3671 > 2017-01-12T07:24 = 123.9212 > 2017-01-12T07:25 = 70.3219 > On Thu, Jan 12, 2017 at 10:49 AM, Stephen Powis <spo...@salesforce.com> wrote: > Just realized that GCEasy doesn't keep reports around for very long

Re: Taking a long time to roll a new log segment (~1 min)

scenario, but I'm having trouble finding the long GC pauses described in that ticket in my own logs. On Thu, Jan 12, 2017 at 10:42 AM, tao xiao <xiaotao...@gmail.com> wrote: > You may run into this bug https://issues.apache.org/jira/browse/KAFKA-4614 > > On Thu, 12 Jan 2017 at 23:3

Re: Taking a long time to roll a new log segment (~1 min)

> Can you collect garbage collection stats and verify there isn't a long GC > happening at the same time? > > -Dave > > -Original Message- > From: Stephen Powis [mailto:spo...@salesforce.com] > Sent: Thursday, January 12, 2017 8:34 AM > To: users@kafka.apache.org &g

Re: Taking a long time to roll a new log segment (~1 min)

: > http://kafka.apache.org/documentation.html#os > > -Ewen > > On Mon, Jan 9, 2017 at 12:45 PM, Stephen Powis <spo...@salesforce.com> > wrote: > > > Hey! > > > > I've run into something concerning in our production clusterI believe > &

Taking a long time to roll a new log segment (~1 min)

Hey! I've run into something concerning in our production clusterI believe I've posted this question to the mailing list previously ( http://mail-archives.apache.org/mod_mbox/kafka-users/201609.mbox/browser) but the problem has become considerably more serious. We've been fighting issues

Re: kafka cluster network bandwidth is too high

Yea, we have a 7 node cluster with ~200 topics and see sustained 100Mbps going between the nodes. Very bandwidth hungry :p On Thu, Dec 8, 2016 at 1:51 AM, Matthias J. Sax wrote: > You cannot sent images over the mailing list. They get automatically > removed. > > On

Slow 'rolled new log segment' ?

Hey! I'm trying to determine why I had a weird hiccup in publishing to my Kafka cluster. I'm running Kafka 0.10.0.1. Here's a graph of my producer times: You can see at around ~10:08 we had a large spike in kafka's internal publishing, waiting for the leader.

Re: [DISCUSS] Java 8 as a minimum requirement

+1 On Thu, Jun 16, 2016 at 5:05 PM, Philippe Derome wrote: > I strongly support motion having difficulty running (Apache Kafka as > opposed to Confluent) Stream examples with JDK 8 today. > On 16 Jun 2016 4:46 p.m., "Ismael Juma" wrote: > > > Hi all, > >

Re: JVM Optimizations

around whether > these options (taken from https://kafka.apache.org/081/ops.html) helped? > Would prefer to not just blindly turn on options if possible. > > -XX:PermSize=48m > -XX:MaxPermSize=48m > -XX:MaxGCPauseMillis=20 > -XX:InitiatingHeapOccupancyPercent=35 > > Thanks! > Be

Re: JVM Optimizations

NOTE -- GC tuning is outside the realm of my expertise by all means, so I'm not sure I'd use our info as any kind of benchmark. But in the interest of sharing, we use the following options export KAFKA_HEAP_OPTS="-Xmx12G -Xms12G" > > export KAFKA_JVM_PERFORMANCE_OPTS="-server

Re: Test

Hello! On Sun, May 29, 2016 at 1:02 PM, Igor Kravzov wrote: > Please someone reply. Not sure if subscribed properly. >

Re: Hash partition of key with skew

Having difficulty following the use cases behind this thread so this could be entirely noise... but I'll toss this out there in case its relevant: https://issues.apache.org/jira/browse/KAFKA- We had an issue where a producer was setting the same key on every message but distributing the

Re: Mirror maker Configs 0.9.0

I've attached my two configs here. Pay close attention to the --num-streams argument to mirror-maker. I have a lot of thru-put on my topics so I ended up matching the number of streams = number of partitions for each of my topics. A stream is essentially just a consumer and producer thread. If

Mirrormaker only publishing to a single partition on destination cluster

Hey! I'm using kafka 0.9.0.1 and trying to replicate a cluster from one datacenter to another. mirror-maker properly connects to my source cluster and consumes messages, but for some reason is only publishing to a single partition for my topic in the destination cluster. So all of my partitions

Re: How to bind all Kafka tcp port to private net address

Pretty sure you want to set this option in your server.properties file: # Hostname the broker will bind to. If not set, the server will bind to all > interfaces > #host.name=localhost > On Thu, Jan 28, 2016 at 10:58 PM, costa xu wrote: > My version is kafka_2.11-0.9.0.0. I

Re: Kafka Consumer and Topic Partition

I think you need to have unique consumer group ids for each consumer (if you want both consumers to receive ALL msgs) or multiple partitions setup on your topic with your consumers sharing the same consumer group (each consumer would then get ~half of all the messages) On Thu, Jan 14, 2016 at

Re: How to chose the size of a Kafka broker

I can't really answer your question, but you don't mention your network layout/hardware. May want to add that as a data point in your decision (wouldn't want to overrun your network device(s) on the brokers). On Wed, Jan 13, 2016 at 7:09 PM, Vladoiu Catalin wrote: >

MirrorMaker --whitelist option in 0.9.0.0

Hey! I'm having a little trouble with mirror maker, if I supply --whitelist "*" it will not consume any topics. If do --whitelist "TopicName" it works, for the specified topic. Any idea why my wildcard does not work? Thanks Stephen

Re: Upgraded from 0.8.2 to 0.9.0 and FetchFollower request time has increased

> > On Fri, Jan 8, 2016 at 7:39 AM, Stephen Powis <spo...@salesforce.com> > wrote: > > > Hey! > > > > So I upgraded our production kafka cluster from 0.8.2 to 0.9.0 this > > morning, and it seems like everything went smoothly. > > > > As the las

Re: MirrorMaker --whitelist option in 0.9.0.0

me :/ On Fri, Jan 8, 2016 at 9:54 PM, tao xiao <xiaotao...@gmail.com> wrote: > Mirror maker white list uses Java pattern > https://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html. In > you case the whitelist is ".*" > > On Fri, 8 Jan 2016 at 20:54 Ste

Upgraded from 0.8.2 to 0.9.0 and FetchFollower request time has increased

Hey! So I upgraded our production kafka cluster from 0.8.2 to 0.9.0 this morning, and it seems like everything went smoothly. As the last step when I changed the inter.broker.protocol.version to 0.9.0.0 and did a rolling restart, I noticed that the FetchFollower request times increased fairly

mirror maker against 0.8.2 source cluster and 0.9.0 destination cluster

Hey! So I'm trying to get mirror maker going between two different clusters. My source cluster is version 0.8.2 and my destination cluster is 0.9.0 running the mirror maker code from the 0.9.0 release. Does anyone know if this is possible to do? I'm aware that the protocol changed slightly

Question about what happens when producer timeout occurs

Hello! I'm curious as to what exactly happens in a certain scenario with kafka, and was unable to find it clearly documented on the website anywhere (or I'm just blind). When producing messages into kafka, the producer can specify a timeout parameter (see producer API here: