Since there are 2 volumes, is the second just a
> better update?, or should I read the first one too?.
>
> Thanks in advance,
> Miguel Suárez
--
*Todd Palino*
Senior Staff Engineer, Site Reliability
Capacity Engineering
linkedin.com/in/toddpalino
were perhaps intending to emphasise the Cruise
> Control or Confluents self-balancing-cluster / auto-balancing features were
> preferable, but in my very brief Google didn't see any advice to set
> auto.leader.rebalance.enabled to false to use those tools.
>
> So yeah, just curious
Going to pile on here, and also say take that blog post with a grain of
salt :)
Look at your data size, and your desired degree of parallelism for
consumers, and guide to that. Yes, 1000 partitions takes a while to create.
But unless your data size is something like 1 TB per day or more, you
e help me to configure the topics and partitions.
>
> On Tue, Jan 8, 2019 at 9:19 PM Todd Palino wrote:
>
> > I think you’ll need to expand a little more here and explain what you
> mean
> > by processing them in parallel. Nearly by definition, parallelization and
> > strict o
000 sensor messages in parallel but each sensor
> message should be in order.If I create 1 partition it doesn't give high
> throughput .Order is guaranteed only inside the partition. How can
> parallelize messages without changing the order pls help me to find the
> solution.
>
> ———-
> <http://www.trimble.com/>
> Srinivasa Balaji L
> Principal Architect Cloud & DevOPS - TPaaS
> 10368, Westmoor Drive, Westminster, CO 80021
> *M*: +1(303) 324-9822 <+919790804422>
> *Email*: lsbal...@trimble.com
>
--
*Todd Palino*
Senior Staff Engineer, Site Reliability
Data Infrastructure Streaming
linkedin.com/in/toddpalino
gt; Can I have 20k partition on a single kafka broker ?
>
--
*Todd Palino*
Senior Staff Engineer, Site Reliability
Data Infrastructure Streaming
linkedin.com/in/toddpalino
ays the Github issues, or PRs for contributing!
-Todd
--
*Todd Palino*
Senior Staff Engineer, Site Reliability
Data Infrastructure Streaming
linkedin.com/in/toddpalino
com>
wrote:
> Is it possible to change the replication factor in runtime? We're using
> 10.x version.
>
> Thanks,
> Devendar
>
--
*Todd Palino*
Senior Staff Engineer, Site Reliability
Data Infrastructure Streaming
linkedin.com/in/toddpalino
er
> if MirrorMaker is also in the source data center (near the source Kafka
> cluster), especially if I can't aggregate the metrics from the 2 data
> centers. Is there anything else that would influence me to deploy in
> MirrorMaker in either data center?
>
> Thanks
>
> Vu
a will bufferize logs and elasticsearch will index its.
>
> Is it a bad practice to have all these JVMs on the same virtual machine ?
> What do you recommend (number of machines, quantity of GB, CPU...) ? For
> the moment, each node has 4 vcpu.
>
> Gabriel.
>
> 2017-08-07 1
To avoid swap you should set swappiness to 1, not 0. 1 is a request (don't
swap if avoidable) whereas 0 is a demand (processes will be killed as OOM
instead of swapping.
However, I'm wondering why you are running such large heaps. Most of the ZK
heap is used for storage of the data in memory, and
in the source datacenter as well. How does the OS/broker tuning
> affect those consumers that are close to the source datacenter? Will they
> continue to function well?
>
> -James
>
> > On Jul 23, 2017, at 7:16 AM, Todd Palino <tpal...@gmail.com> wrote:
> >
>
We have 4 topics are white listed to sync from EU to US. Only one of them
> > is high throughput. We also have a message handler to strip off some
> > sensitive information from EU to US but it only works on a low thru put
> > topic; the message handler still try to process the other topics but let
> it
> > pass thru.
> >
> > Thanks,
> > Sunil Parmar
>
--
*Todd Palino*
Senior Staff Engineer, Site Reliability
Data Infrastructure Streaming
linkedin.com/in/toddpalino
es and
> > > subsidiaries that is proprietary, privileged, confidential and/or
> subject
> > > to copyright. Any review, retransmission, dissemination or other use
> of,
> > or
> > > taking of any action in reliance upon, this information by persons or
> > > entities other than the intended recipient(s) is prohibited and may be
> > > unlawful. If you received this in error, please contact the sender
> > > immediately and delete and destroy the communication and all of the
> > > attachments you have received and all copies thereof.
> > >
> > >
> > >
> >
>
--
*Todd Palino*
Senior Staff Engineer, Site Reliability
Data Infrastructure Streaming
linkedin.com/in/toddpalino
h,
> *
>
> *kafka-reassign-partition.sh)*
> *One of the blogs mentioned that - it is preferable to Re-balance Kafka
> topics manually, since setting *
>
> *auto.leader.rebalance.enable = true causes issues.*
>
> Pls let me know.
> Any other best practices wrt. Re-bal
.example.com
> > #zookeeper-port=2181
> > #zookeeper-path=/kafka-cluster/stormconsumers
> >
> > [tickers]
> > broker-offsets=20
> >
> > [lagcheck]
> > intervals=10
> > expire-group=604800
> >
> > [notify]
> > interval=10
> >
> > [httpserver]
> > server=on
> > port=8000
> > ; Alternatively, use listen (cannot be specified when port is)
> > ; listen=host:port
> > ; listen=host2:port2
> >
> > [smtp]
> > server=mailserver.example.com
> > port=25
> > from=burrow-nore...@example.com
> > template=config/default-email.tmpl
> >
> > [emailnotifier "b...@example.com"]
> > group=local,critical-consumer-group
> > group=local,other-consumer-group
> > interval=60
> >
> > [notify]
> > interval=10
> >
> > [httpnotifier]
> > url=http://notification.server.example.com:9000/v1/alert
> > interval=60
> > extra=app=burrow
> > extra=tier=STG
> > template-post=config/default-http-post.tmpl
> > template-delete=config/default-http-delete.tmpl
> > timeout=5
> > keepalive=30
> >
> > So Can you please let me know what I am missing and how to fix these
> > issues.Any help would be appreciated.
> >
> >
> >
> > Regards,
> > Abhimanyu
> >
>
--
*Todd Palino*
Senior Staff Engineer, Site Reliability
Data Infrastructure Streaming
linkedin.com/in/toddpalino
cation factor to 3 for those topics now? Do we need to come up
> with a manual assignment plan for each of the partitions? Is there any
> quicker way to achieve this?
>
> Thanks!
>
--
*Todd Palino*
Senior Staff Engineer, Site Reliability
Data Infrastructure Streaming
linkedin.com/in/toddpalino
. Right now I have to restart the broker to
> failover it to other brokers. Is there a way to failover controller to a
> specific broker? Is there a way to failover it without restart the broker?
>
> Thanks,
> Jun
--
*Todd Palino*
Senior Staff Engineer, Site Reliability
Data Infrast
ants to read
> it,
> > it simply reads at the right offset...
> >
> > Also I m wondering if it s the case if we don t use keys (pure queuing
> > system with key=null).
> >
> > Cheers
> > Nico
> >
> >
>
--
*Todd Palino*
Staff Site Reliability Engineer
Data Infrastructure Streaming
linkedin.com/in/toddpalino
is
> e-mail in error, please notify the sender by reply e-mail immediately and
> destroy all copies of the e-mail and any attachments.
>
>
--
*Todd Palino*
Staff Site Reliability Engineer
Data Infrastructure Streaming
linkedin.com/in/toddpalino
gest no, but doesn't
> address the point directly:
> http://events.linuxfoundation.org/sites/events/files/slides/
> Kafka%20At%20Scale.pdf
>
> --
> Jack Foy <j...@hiya.com>
>
--
*Todd Palino*
Staff Site Reliability Engineer
Data Infrastructure Streaming
linkedin.com/in/toddpalino
you try min.insync.replicas=2 in server.properties and acks=all on
> producer? min.insync.replicas only applies when acks=all.
>
> -James
>
> >
> > -Original Message-
> > From: Todd Palino [mailto:tpal...@gmail.com]
> > Sent: Monday, March 06, 2
nts, or the taking of any unauthorized
> action based on information contained herein is strictly prohibited.
> Unauthorized use of information contained herein may subject you to civil
> and criminal prosecution and penalties. If you are not the intended
> recipient, please immediate
gt; https://github.com/ktls/af_ktls
> http://www.phoronix.com/scan.php?page=news_item=FreeBSD-Faster-Sendfile
>
> Ismael
>
> On Mon, Mar 6, 2017 at 4:18 PM, Todd Palino <tpal...@gmail.com> wrote:
>
> > So that’s not quite true, Hans. First, as far as the performance hit
> it,
> > it simply reads at the right offset...
> >
> > Also I m wondering if it s the case if we don t use keys (pure queuing
> > system with key=null).
> >
> > Cheers
> > Nico
>
>
--
*Todd Palino*
Staff Site Reliability Engineer
Data Infrastructure Streaming
linkedin.com/in/toddpalino
et...
>
> Also I m wondering if it s the case if we don t use keys (pure queuing
> system with key=null).
>
> Cheers
> Nico
>
--
*Todd Palino*
Staff Site Reliability Engineer
Data Infrastructure Streaming
linkedin.com/in/toddpalino
or now, I am doing “kill –9 processID” to stop the Kafka Mirror
> Maker. I am wondering whether there is a better way (e.g. a command) to do
> so? I don’t expect to stop the mirror maker frequently but I would like to
> have a script to automate the start and stop.
>
> Thanks a lot
; Also - what do you set openfiles to? I have it at 65535 but I just read a
> doc that suggested > 100K is better
>
>
> On Tue, Feb 21, 2017 at 10:45 AM, Todd Palino <tpal...@gmail.com> wrote:
>
> > So I think the important thing to look at here is the IO wait on your
> &g
hines)
> it would appear that for our message size we are at the max. This would
> argue that we need to shrink the message size - so perhaps switching to
> avro is the next step?
>
--
*Todd Palino*
Staff Site Reliability Engineer
Data Infrastructure Streaming
linkedin.com/in/toddpalino
on reassignment:
> Reassignment of partition [foo,2] is still in progress
>
> And, I would think it would since 1001 is already an ISR and the leader is
> already 1004.
>
> How do I resolve this?
>
--
*Todd Palino*
Staff Site Reliability Engineer
Data Infrastructure Streaming
linkedin.com/in/toddpalino
the servers. It’s worked for the last couple without a problem.
-Todd
On Tue, Dec 20, 2016 at 7:55 PM, Sanjeev T <san...@gmail.com> wrote:
> Hi,
>
> Can some of you share points on, the versions and handling leap second
> delay on Dec 31, 2016.
>
> Regards
> -Sanjeev
&
the same problem.
> nvalid receive (size = 1347375956 larger than 104857600).
>
> When trying to increase the size, Java Out of Memory Exception.
> Did you find a work around for the same ??
>
> Thanks.
>
--
*Todd Palino*
Staff Site Reliability Engineer
Data Infrastruc
he topic.
> >
> > -- Surendra Manchikanti
> >
> > On Sat, Dec 10, 2016 at 10:59 AM, Todd Palino <tpal...@gmail.com> wrote:
> >
> > > Are you running something else besides the consumers that would
> maintain
> > a
> > > memory of the to
recreating a topic that has been deleted as it issues a metadata request to
try and find out what happened after an offset request for the topic fails.
-Todd
On Fri, Dec 9, 2016 at 8:37 AM, Tim Visher <tim.vis...@gmail.com> wrote:
> On Fri, Dec 9, 2016 at 11:34 AM, Todd Palino <tpal.
nder in
> > > zookeeper,
> > > > and then brought kafka back up.
> > > >
> > > > After doing all that, the topic comes back, every time.
> > > >
> > > > What can we do to delete that topic?
> > > >
> > > > --
> > > >
> > > > In Christ,
> > > >
> > > > Timmy V.
> > > >
> > > > http://blog.twonegatives.com/
> > > > http://five.sentenc.es/ -- Spend less time on mail
> > > >
> > >
> >
>
--
*Todd Palino*
Staff Site Reliability Engineer
Data Infrastructure Streaming
linkedin.com/in/toddpalino
; Some other hand-wavy ideas that came to mind might be:
> * handling topic creation in a MirrorMakerMessageHandler
> * handling topic creation in an interceptor
>
> Anyway, was hoping to get some thoughts from people who are already doing
> this.
>
> Thanks!
> -James
>
>
--
*Todd Palino*
Staff Site Reliability Engineer
Data Infrastructure Streaming
linkedin.com/in/toddpalino
6 brokers. Now one of the brokers (B-16) got completely messed up
> and is sent for repair.
>
> But I can still see some partitions including the B-16 in its replicas,
> thereby becoming under-replicated.
>
> Is there a proper way to take broker out of rotation?
>
> Praveen
itten in the past trying to automate
> legitimate mail to GMail. Getting washed out with the spammers is a real
> hazard.
>
> Thanks!
>
>
>
> --
> information informs. analysis enlightens.
>
> Brian M. Dennis, PhD
> 43403 Coton Commons Dr, Leesburg, VA 20176
> e:
tion as I do for the other topics? Is it safe to reassign the new
> layout for this internal topic, using kafka-reassign-partitions.sh?
>
>
> Thanks, Anderson
>
--
*Todd Palino*
Staff Site Reliability Engineer
Data Infrastructure Streaming
linkedin.com/in/toddpalino
ummy-voidbridge-oneworks-dummy-1_/10.100.0.113
> voidbridge-oneworks-dummy integration-oneworks-dummy 15
> 0 0 0
> integration-oneworks-dummy-voidbridge-oneworks-dummy-1_/10.100.0.113
>
> > On 8 Jul 2016, at 17:20, Tom Dearman <t
partitions with something in them. I know this is not
> very like production, but on my local this I was only testing with one user
> so get just one partition filled.
>
> Tom
> > On 6 Jul 2016, at 18:08, Todd Palino <tpal...@gmail.com <javascript:;>>
> wrote:
>
r will be
> consuming data over the internet. Is there a way to encrypt this data
> without turning SSL on. There would be some performance degradation and it
> is not required in other scenarios. Has anyone tried doing cross-regional
> replication in production?
>
> Thanks,
&g
ion=xxx
>
> On Wed, Jul 6, 2016 at 5:36 PM, Todd Palino <tpal...@gmail.com
> <javascript:;>> wrote:
> > We do this through our monitoring agents by pulling it as a metric from
> the
> > LogEndOffset beans. By putting it into our metrics system we get a
> mappin
my colleague Jon Bringhurst
profusely for helping to get the structure around the project and the
documentation cleaned up.
-Todd
--
*Todd Palino*
Staff Site Reliability Engineer
Data Infrastructure Streaming
linkedin.com/in/toddpalino
--
*Todd Palino*
Staff Site Reliability Engineer
Data
--new-consumer --bootstrap-server
> 10.10.1.61:9092 --describe —group consumer_group_name and parsing the
> response. Is it safe or advisable to do this? I like the fact that it
> tells me each partition lag, although it is also not available if no
> consumer from the group is currently co
ybe there is a better, more efficient, way to do it?
>
> Cheers,
> -Kristoffer
>
> [1] https://gist.github.com/krisskross/a49e462bedb89505e372672cd81129ab
>
--
*Todd Palino*
Staff Site Reliability Engineer
Data Infrastructure Streaming
linkedin.com/in/toddpalino
oku Kafka
> >
> >On Tuesday, 14 June 2016, Mudit Kumar <mudit.ku...@askme.in> wrote:
> >
> >> Hey,
> >>
> >> How can I delete particular messages from particular topic?Is that
> >> possible?
> >>
> >> Thanks,
> >> Mudit
> >>
> >>
>
>
--
*Todd Palino*
Staff Site Reliability Engineer
Data Infrastructure Streaming
linkedin.com/in/toddpalino
and thinking about adding a few more.
> The
> > new broker machines would have a lot more storage available to them than
> > the existing brokers. Am I setting myself up for operational headaches by
> > deploying a heterogeneous (in terms of storage capacity) cluster?
> >
>
> > >
> > > > > > >
> > > > > > > Is this the confluent doc you were referring to?
> > > > > > > https://protect-us.mimecast.com/s/arXXBOspkvORCD
> > > > > > >
> > > > > > > Thanks!
> > > > > > >
> > > > > > > Lawrence Weikum
> > > > > > >
> > > > > > >
> > > > > > > On 6/9/16, 1:32 PM, "Tom Crayford" <tcrayf...@heroku.com>
> wrote:
> > > > > > >
> > > > > > > >Hi Lawrence,
> > > > > > > >
> > > > > > > >What JVM options were you using? There's a few pages in the
> > > > confluent
> > > > > docs
> > > > > > > >on JVM tuning iirc. We simply use the G1 and a 4GB Max heap
> and
> > > > things
> > > > > > > work
> > > > > > > >well (running many thousands of clusters).
> > > > > > > >
> > > > > > > >Thanks
> > > > > > > >Tom Crayford
> > > > > > > >Heroku Kafka
> > > > > > > >
> > > > > > > >On Thursday, 9 June 2016, Lawrence Weikum <
> lwei...@pandora.com>
> > > > > wrote:
> > > > > > > >
> > > > > > > >> Hello all,
> > > > > > > >>
> > > > > > > >> We’ve been running a benchmark test on a Kafka cluster of
> ours
> > > > > running
> > > > > > > >> 0.9.0.1 – slamming it with messages to see when/if things
> > might
> > > > > break.
> > > > > > > >> During our test, we caused two brokers to throw OutOfMemory
> > > errors
> > > > > > > (looks
> > > > > > > >> like from the Heap) even though each machine still has 43%
> of
> > > the
> > > > > total
> > > > > > > >> memory unused.
> > > > > > > >>
> > > > > > > >> I’m curious what JVM optimizations are recommended for Kafka
> > > > > brokers?
> > > > > > > Or
> > > > > > > >> if there aren’t any that are recommended, what are some
> > > > > optimizations
> > > > > > > >> others are using to keep the brokers running smoothly?
> > > > > > > >>
> > > > > > > >> Best,
> > > > > > > >>
> > > > > > > >> Lawrence Weikum
> > > > > > > >>
> > > > > > > >>
> > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Dustin Cote
> > > confluent.io
> > >
> >
>
>
>
> --
> Dustin Cote
> confluent.io
>
--
*Todd Palino*
Staff Site Reliability Engineer
Data Infrastructure Streaming
linkedin.com/in/toddpalino
dstk...@gmail.com> wrote:
> > >
> > > > Hi All,
> > > >
> > > > Does anyone have any experience of using kafka behind a load
> balancer?
> > > >
> > > > Would this work? Are there any reasons why you would not want to do
> it?
> > > >
> > > > Thanks!
> > > >
> > >
> >
>
>
>
> --
> Thanks,
> Ewen
>
--
*Todd Palino*
Staff Site Reliability Engineer
Data Infrastructure Streaming
linkedin.com/in/toddpalino
t would be nice if it would. I don't know if there
> is anything in log file format info or Zoo info attached to original topic
> which would prevent it to work.
>
> Petr
>
> -----Original Message-
> From: Todd Palino [mailto:tpal...@gmail.com]
> Sent: 1. června 2016 10:48
sible to change an existing topic name?
>
> Thanks and best wishes
> Johannes
>
--
*Todd Palino*
Staff Site Reliability Engineer
Data Infrastructure Streaming
linkedin.com/in/toddpalino
. Yes, you can certainly use a wildcard consumer,
but if you’re not doing that you have to maintain some mapping of consumers
to topics. And if you are using a wildcard consumer, you’re going to run
into issues with the number of topics any given group is consuming at some
point. Your system may wo
avoided to optimise for this level of traffic
>
> To help your answer, we're looking at potentially 16GBit/sec inbound which
> concerns our network team.
>
> If you can please share pointers to existing materials or specific details
> of your deployment, that will be great.
&g
16, at 10:33 AM, Todd Palino <tpal...@gmail.com> wrote:
> >
> > There’s two things that people usually mean when they talk about
> > rebalancing.
> >
> > One is leader reelection, or preferred replica election, which is
> sometimes
> > confus
e to configure the re-balancing option?
>
> Regards,
> Srikanth
>
--
*—-*
*Todd Palino*
Staff Site Reliability Engineer
Data Infrastructure Streaming
linkedin.com/in/toddpalino
cluster? Should I increase or decrease
> partitions? What can be done to increase the amount of logs being read
> from the cluster and ingested into Elastisearch?
>
> Like I said, very new to kafka.
>
> Thanks for the help
> Tim
>
--
*—-*
*Todd Palino*
Staff Site Reliability Engineer
Data Infrastructure Streaming
linkedin.com/in/toddpalino
> Backend engineer
> Tink AB
>
> Email: jens.ran...@tink.se
> Phone: +46 708 84 18 32
> Web: www.tink.se
>
> Facebook <https://www.facebook.com/#!/tink.se> Linkedin
> <
> http://www.linkedin.com/company/2735919?trk=vsrp_companies_res_photo=VSRPsearchId%3A1057023
r is that "it depends on load", but would be interested
> > in any opinions on this anyway.
> >
> > Thanks!
> > \EF
> >
>
--
*—-*
*Todd Palino*
Staff Site Reliability Engineer
Data Infrastructure Streaming
linkedin.com/in/toddpalino
.
-Todd
On Saturday, December 19, 2015, Akhilesh Pathodia <
pathodia.akhil...@gmail.com> wrote:
> What is the process for deleting the consumer group from zookeeper? Should
> I export offset, delete and then import?
>
> Thanks,
> Akhilesh
>
> On Fri, Dec 18, 2015 at
manager where I can delete the group?
>
> Thanks
>
> On Sat, Dec 19, 2015 at 11:47 PM, Todd Palino <tpal...@gmail.com> wrote:
>
> > If what you want to do is reset to smallest, all you need to do is stop
> the
> > consumer, delete the group from Zookeeper, and res
et the kafka property kafka.auto.offset.reset to "smallest",
> but it does not reset the offset in zookeeper and that's why flume will not
> read messages from first offset.
>
> Is there any way to reset kafka offset in zookeeper?
>
> Thanks,
> Akhilesh
>
--
gt; > Thanks,
> > > Akhilesh
> > >
> >
> >
> > --
> > Jens Rantil
> > Backend engineer
> > Tink AB
> >
> > Email: jens.ran...@tink.se
> > Phone: +46 708 84 18 32
> > Web: www.tink.se
> >
> > Facebook <https://www.facebook.com/#!/tink.se> Linkedin
> > <
> >
> http://www.linkedin.com/company/2735919?trk=vsrp_companies_res_photo=VSRPsearchId%3A1057023381369207406670%2CVSRPtargetId%3A2735919%2CVSRPcmpt%3Aprimary
> > >
> > Twitter <https://twitter.com/tink>
> >
>
--
*—-*
*Todd Palino*
Staff Site Reliability Engineer
Data Infrastructure Streaming
linkedin.com/in/toddpalino
> | Contact
> <http://sematext.com/about/contact.html>
>
> On Fri, Dec 18, 2015 at 6:29 PM, Todd Palino <tpal...@gmail.com> wrote:
>
> > That works if you want to set to an arbitrary offset, Marko. However in
> the
> > case the OP described, wanting
hought. Is
> it normal to have a topic with 1000 partitions? I was thinking about about
> two/four partitions per node. is it wrong my thought?
>
> As I'm going to process data with Spark, I could have numberPartitions
> equals numberExecutors in Spark as max, always thinking in the f
Replicas and Isr are both a comma separated list of broker IDs. So in this
output, I am seeing that you have two Kafka brokers with IDs 1 and 2. You
have a topic, capture, with 16 partitions at replication factor 1 (1
replica per partition). The broker with ID 2 is not online, which is why it
mmon.serialization.StringSerializer"
>
>
>
>
> kafkaProducer.send(new ProducerRecord[String,String](topic, key, data),new
> Callback {
> def onCompletion(recordMetadata: RecordMetadata, e: Exception):Unit = {
>if(e != null) {
> logger.error(s"Cou
I’m not quite sure why you would need to do this - the broker IDs are not
significant outside of the internal metadata. But this is what you would
have to do for each move (assuming you are running with at least
replication factor 2):
1) Shut down the broker
2) Clear its partition data
3)
We use loadbalancers for our producer configurations, but what you need to
keep in mind is that that connection is only used for metadata requests.
The producer queries the loadbalancer IP for metadata for the topic, then
disconnects and reconnects directly to the Kafka brokers for producing
Actually, be very careful with this. There are two different things stored
in Zookeeper, and depending on what you're interested in you want to make
sure you're looking at the right one.
If you want to know the replica assignment - that is, what brokers a given
partition is assigned to - you need
(you can use partition reassignment to
change it). But if they are not all the same, some of the tooling will
break (such as altering the partition count for the topic).
-Todd
On Fri, Oct 16, 2015 at 5:39 PM, Todd Palino <tpal...@gmail.com> wrote:
> Actually, be ver
We've had no problems with G1 in all of our clusters with varying load
levels. I think we've seen an occasional long GC here and there, but
nothing recurring at this point.
What's the full command line that you're using with all the options?
-Todd
On Wed, Oct 14, 2015 at 2:18 PM, Scott Clasen
Yes. As long as you have not reassigned the partitions to other brokers,
the wiped broker will rebuild from replication. Keep in mind, however, that
if you are using the retention by time configuration, you will have 2x
retention on that broker for the length of retention. This means that if
your
To answer the question, yes, it is incorrect. There are a few things you
can do to minimize problems. One is to disable unclean leader election, use
acks=-1 on the producers, have an RF of 3 or greater, and set the min ISR
to 2. This means that the topic will only be available if there are at
Multiple topics is the model I would recommend for what you have described.
LinkedIn has an environment where we have a wide mix, in a lot of different
clusters. We have some topics that have one producer and one consumer
(queuing). We have some topics that are multi-producer (tracking and
> Jason
>
> On Fri, Sep 25, 2015 at 12:15 AM, Todd Palino <tpal...@gmail.com> wrote:
>
> > For now, that's the way it is. Historically, we've only monitored the lag
> > for our infrastructure applications. Other users are responsible for
> their
> > own check
for hours.
>
> We also still have developers creating topics semi-regularly, which it
> seems like can cause the high level consumer to disconnect?
>
>
> On Fri, Sep 25, 2015 at 6:16 PM Todd Palino <tpal...@gmail.com
> <javascript:_e(%7B%7D,'cvml','tpal...@gmail.com');>
I don't see the logs attached, but what does the GC look like in your
applications? A lot of times this is caused (at least on the consumer side)
by the Zookeeper session expiring due to excessive GC activity, which
causes the consumers to go into a rebalance and change up their connections.
gt; Could it be the way we are bring up multiple consumers at the same time is
> hitting some sort of endless rebalance cycle? And/or the resulting
> thrashing is causing them to time out, rebalance, etc.?
>
> I've tried attaching the logs again. Thanks!
>
> On Fri, Sep 25, 2015 at
applications for Kafka), but we're not encouraging other internal users to
switch over just yet.
-Todd
On Wed, Sep 23, 2015 at 3:21 PM, James Cheng <jch...@tivo.com> wrote:
>
> On Sep 18, 2015, at 10:25 AM, Todd Palino <tpal...@gmail.com> wrote:
>
> > I think th
moved
over to Kafka committed offsets at this point.
Of course, just those apps do cover well over a hundred consumer groups :)
-Todd
On Thursday, September 24, 2015, James Cheng <jch...@tivo.com> wrote:
>
> > On Sep 24, 2015, at 8:11 PM, Todd Palino <tpal...@gmail.com
>
Retention is going to be based on a combination of both the retention and
segment size settings (as a side note, it's recommended to use
log.retention.ms and log.segment.ms, not the hours config. That's there for
legacy reasons, but the ms configs are more consistent). As messages are
received by
Yes, this is a known concern, and it should be fixed with recent commits.
In the meantime, you'll have to do a little manual cleanup.
The problem you're running into is a corrupt message in the offsets topic.
We've seen this a lot. What you need to do is set the topic configuration
to remove the
the data directory and
> restart the servers. The log indicates all is well.
>
> Do you know what version the fix to this will be in? I'm not looking
> forward to dealing with this on a reoccurring basis.
>
> -John
>
> On Fri, Sep 18, 2015 at 8:48 AM Todd Palino <tpal...
Consumer offsets in Zookeeper are not handled by the Kafka brokers at all -
the consumer writes those directly to Zookeeper. Most likely, what you are
seeing is the interval over which the consumer is committing offsets.
Assuming that you are using the auto.commit.enable setting (it defaults to
This message is regarding a normal connection close. You see it in the logs
for any connection - consumers, producers, replica fetchers. It can be
particularly noisy because metadata requests often happen on their own
connection.
The log message has been moved to debug level in recent commits (it
I put an answer to this on Stack Overflow. Basically, that's not how RF
works for Kafka. It's not a guarantee, it's just how the partitions are
created, and how it is reported when something is down (under replicated
partitions). While there is an option to do auto leader rebalancing,
there's no
nd setups, without finding the
> main cause.
> Can this be a Kernel (version/configuration) or Java(7) issue?
>
> Cheers
> Jörg
>
>
>
> On 04.09.2015 20:24, Todd Palino wrote:
>
>> Jun's post is a good start, but I find it's easier to talk in terms of
>
Jun's post is a good start, but I find it's easier to talk in terms of more
concrete reasons and guidance for having fewer or more partitions per topic.
Start with the number of brokers in the cluster. This is a good baseline
for the minimum number of partitions in a topic, as it will assure
You can use the emailer config in Burrow to send alerts directly (it will
monitor specific groups and send emails out when there is a problem). If
you need something more complex than that, I think the best practice is
always to send the output into an general alert/notification system.
-Todd
On
What Gwen said :)
We developed a python web service internally called Menagerie that provides
this functionality for both Kafka and Zookeeper. We use it to drive a web
dashboard for stats, our (old style) lag checking, and some other CLI
tools. Unfortunately it ties into too much internal
It looks like you did an unclean shutdown of the cluster, in which case
each open log segment in each partition needs to be checked upon startup.
It doesn't really have anything to do with RF=3 specifically, but it does
mean that each of your brokers has 6000 partitions to check.
What is the
To make sure you have a complete answer here, the order of the replica list
that you specify in the partition reassignment will affect the leader
selection, but if the current leader is in the new replica list, it will
not change the leadership to change.
That is, if your current replica list is
That's exactly right. We've been talking about this internally at LinkedIn, and
how to solve it. I think the best option would be to have the broker throw an
error on offset commits until there are enough brokers to fulfill the
configured RF.
We've seen this several times now when
Yes, we use ZK 3.4.6 exclusively at LinkedIn and there's no problem.
-Todd
On Jul 22, 2015, at 9:49 AM, Adam Dubiel dubiel.a...@gmail.com wrote:
Hi,
I don't think it matters much which version of ZK will you use (meaning
minor/patch versions). We have been using 3.4.6 for some time and
This is interesting. We have seen something similar internally at LinkedIn
with one particular topic (and Avro schema), and only once in a while.
We've seen it happen 2 or 3 times so far. We had chalked it up to bad
content in the message, figuring that the sender was doing something like
sending
, Christofer Hedbrandh
christo...@knewton.com wrote:
Todd, the Kafka problems started when one of three ZooKeeper nodes was
restarted.
On Thu, Jul 9, 2015 at 12:10 PM, Todd Palino tpal...@gmail.com wrote:
Did you hit the problems in the Kafka brokers and consumers during the
Zookeeper
Did you hit the problems in the Kafka brokers and consumers during the
Zookeeper problem, or after you had already cleared it?
For us, we decided to skip the leap second problem (even though we're
supposedly on a version that doesn't have that bug) by shutting down ntpd
everywhere and then
Yes and no. We're running a version about a month behind trunk at any given
time here at LinkedIn. That's generally the amount of time we spend testing
and going through our release process internally (less if there are no
problems). So it can be done.
That said, we also have several Kafka
1 - 100 of 160 matches
Mail list logo
