Hi everyone,
I wanted to share with you some common utilities we open sourced (
http://engineering.cerner.com/blog/cerner-open-sources-its-kafka-utilities/).
We hope others find them useful.
Bryan
Hi everyone,
We've seen some instances of our consumer groups when running normally not
process any messages from some partitions for minutes while other
partitions are seeing regularly updates in seconds. In some cases when a
consumer group had a significant lag (hours of messages), some
I'm interested in knowing if theres any plan or idea to add transactions to
connect.
We make use of the JDBC source connector and its bulk extract mode. It
would be great if the connector could create a transaction around the
entire extraction in order to ensure the entire table's data made it
s just 3 seconds, not minutes though, right? Would
>> you have any logs to share?
>>
>> Thanks
>> Eno
>> > On 14 Jun 2017, at 16:14, Bryan Baugher <bjb...@gmail.com> wrote:
>> >
>> > Our consumer group isn't doing anything state
ogs to share?
>
> Thanks
> Eno
> > On 14 Jun 2017, at 16:14, Bryan Baugher <bjb...@gmail.com> wrote:
> >
> > Our consumer group isn't doing anything stateful and we've seen this
> > behavior for existing groups as well. It seems like timing could be an
> >
ated to
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-134%3A+Delay+initial+consumer+group+rebalance
> and https://issues.apache.org/jira/browse/KAFKA-4925.
>
> -James
>
> > On Jun 13, 2017, at 7:02 AM, Bryan Baugher <bjb...@gmail.com> wrote:
> >
&g
The topics already exist prior to starting any of the consumers
On Mon, Jun 12, 2017 at 9:35 PM J Pai <jai.forums2...@gmail.com> wrote:
> When are the topics on which these consumer groups consume, created?
>
> -Jaikiran
> On 13-Jun-2017, at 3:18 AM, Bryan Baugher <bj
Hi everyone,
We are currently experiencing slow startup times for our consumer groups
(16-32 processes for a hundred or more partitions) in the range of minutes
(3-15 minutes), where little to no messages are consumed before suddenly
everything just starts working at full speed.
I'm currently
Hi everyone,
I've been trying out Kafka security and was curious if there were
plans/issues to add wildcards in resources or user group support to Kafka
ACLs? If they are already implemented point me to the doc on how to use it?
Specifically it would be nice to give group's of users access to
This can potentially cause the FATAL error. We do plan to
> fix KAFKA-1211 in the future, but it may take some time.
>
> Thanks,
>
> Jun
>
> On Fri, Aug 26, 2016 at 6:53 AM, Bryan Baugher <bjb...@gmail.com> wrote:
>
> > We didn't suffer any data loss nor was there an
We didn't suffer any data loss nor was there any power outage that I know
of.
On Fri, Aug 26, 2016 at 5:14 AM Khurrum Nasim <khurrumnas...@gmail.com>
wrote:
> On Tue, Aug 23, 2016 at 9:00 AM, Bryan Baugher <bjb...@gmail.com> wrote:
> >
> > > Hi everyone,
>
Hi everyone,
Yesterday we had lots of network failures running our Kafka cluster
(0.9.0.1 ~40 nodes). We run everything using the higher durability settings
in order to avoid in data loss, producers use all/-1 ack, topics/brokers
have min insync replicas = 2. unclean leader election = false, and
Found I still hit this issue without VPN. I had to make the cluster's user
a super user or at least give it appropriate privileges
On Thu, Aug 4, 2016 at 11:39 AM Bryan Baugher <bjb...@gmail.com> wrote:
> Figured this out. This had to do with me being on a VPN and running
> everyt
Figured this out. This had to do with me being on a VPN and running
everything locally
On Thu, Aug 4, 2016 at 11:26 AM Bryan Baugher <bjb...@gmail.com> wrote:
> I managed to replicate this issue using the default provided config for
> kafka/zookeeper in the binary artifact and using
Can you enable Authorization debug logs and check for logs related to
> denied operations..
> we should also enable operations on Cluster resource.
>
>
> Thanks,
> Manikumar
>
> On Thu, Aug 4, 2016 at 1:51 AM, Bryan Baugher <bjb...@gmail.com> wrote:
>
> > Hi everyon
Hi everyone,
I was trying out kerberos on Kafka 0.10.0.0 by creating a single node
cluster. I managed to get everything setup and past all the authentication
errors but whenever I try to use the console producer I get 'Error while
fetching metadata ... LEADER_NOT_AVAILABLE'. In this case I've
Hi everyone,
With the new Kafka consumer[1] is it possible to use zookeeper based offset
storage?
Bryan
[1] -
http://kafka.apache.org/0100/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html
Hi everyone,
I'm currently experiencing a situation that looks very similar to this[1].
I'm running Kafka 0.8.2.1 and we use it with Storm but don't use Storm's
kafka spout since it doesn't quite work for our needs. I've been able to
reproduce the issue locally by creating a storm topology with 1
Hi everyone,
We recently switched to round robin partition assignment after we noticed
that range partition assignment (default) will only make use of the first X
consumers were X is the number of partitions for a topic our consumers are
interested in. We then noticed the caveat in round robin,
Hi everyone,
We are running Kafka 0.8.1.1 with Storm. We wrote our own spout which uses
the high level consumer API. Our setup is to create 4 spouts per worker. If
your not familiar with Storm its basically 4 kafka consumers per java
process. This particular consumer group is interested in 20
:13 AM Bryan Baugher bjb...@gmail.com wrote:
Hi everyone,
We are running Kafka 0.8.1.1 with Storm. We wrote our own spout which uses
the high level consumer API. Our setup is to create 4 spouts per worker. If
your not familiar with Storm its basically 4 kafka consumers per java
process
Sure, https://issues.apache.org/jira/browse/KAFKA-2114
On Sat, Apr 11, 2015 at 3:16 PM Gwen Shapira gshap...@cloudera.com wrote:
this seems like a bug. We expect broker settings to set
defaults for topics. Perhaps open a JIRA?
On Fri, Apr 10, 2015 at 1:32 PM, Bryan Baugher bjb
To answer my own question via testing, setting min.insync.replicas on the
broker does not change the default. The only way I can find to change this
is by editing the topic config.
On Fri, Apr 10, 2015 at 12:57 PM Kamal C kamaltar...@gmail.com wrote:
Similarly my producers are currently
Hi everyone,
I'm starting to look at how you might upgrade your cluster even if a major
upgrade with non-passive API changes and do this all in uptime (i.e. do not
read/write). I realize this may involve knowledge in how Kafka is used, in
our case we own all the reading and writing for now.
Has
Hi everyone,
I'm looking for the API to create/manager topics. Is this class[1] the one
I should be using? Also what if I'm wanting to create the topic
synchronously?
[1] -
https://github.com/apache/kafka/blob/0.8.1.1/core/src/main/scala/kafka/admin/AdminUtils.scala#L149-L157
Bryan
21, 2014 at 3:28 PM, Guozhang Wang wangg...@gmail.com wrote:
Bryan,
Did you take down some brokers in your cluster while hitting KAFKA-1028? If
yes, you may be hitting KAFKA-1647 also.
Guozhang
On Mon, Oct 20, 2014 at 1:18 PM, Bryan Baugher bjb...@gmail.com wrote:
Hi everyone,
We run
Hi everyone,
We run a 3 Kafka cluster using 0.8.1.1 with all topics having a replication
factor of 3 meaning every broker has a replica of every partition.
We recently ran into this issue (
https://issues.apache.org/jira/browse/KAFKA-1028) and saw data loss within
Kafka. We understand why it
Hi,
We have been trying out the kafka 0.8.0 beta1 for awhile and recently
attempted to upgrade to 0.8.0 but noticed that the stop server script
doesn't seem to stop the broker anymore. I noticed here[1] that a commit
was made before the release to change the signal sent to stop the broker
from
RHEL 6.4 64bit
Java 6u35
On Tue, Dec 17, 2013 at 10:57 PM, Jun Rao jun...@gmail.com wrote:
Which OS are you on?
Thanks,
Jun
On Tue, Dec 17, 2013 at 11:15 AM, Bryan Baugher bjb...@gmail.com wrote:
Hi,
We have been trying out the kafka 0.8.0 beta1 for awhile and recently
29 matches
Mail list logo
