Re: resilient producer
+1 how about posting yours to GitHub? Sounds like a good contrib project. Sent from my iPhone On Jan 15, 2013, at 12:29 PM, Stan Rosenberg wrote: > Hi, > > In out current data ingestion system, producers are resilient in the sense > that if data cannot be reliably published (e.g., network is down), it is > spilled onto local disk. > A separate process runs asynchronously and attempts to publish spilled > data. I am curious to hear what other people do in this case. > Is there a plan to have something similar integrated into kafka? (AFAIK, > current implementation gives up after a configurable number of retries.) > > Thanks, > > stan
Re: Best practices for changing partition numbers
David Sounds like there is an admin ddl you have to run on your existing topic to change the # after creation. Sent from my iPhone On Jan 8, 2013, at 2:24 PM, David Ross wrote: > Yeah that makes sense, but what if we do need to change the number of > partitions? What if we need to reduce it? > > On Tue, Jan 8, 2013 at 12:42 PM, Jun Rao wrote: > >> If you don't have a lot of topics, one thing you can do is to >> over-partition a topic. >> >> Also, in 0.7, # of partitions grows with brokers. This is going to change >> in 0.8, in which # of partitions is specified at topic creation time and >> won't change as brokers change. One needs to use an admin DDL to change # >> of partitions. >> >> Thanks, >> >> Jun >> >> On Mon, Jan 7, 2013 at 10:23 PM, David Ross wrote: >> >>> Hello, >>> >>> We have found that, for our application, having a number of total >>> partitions as a multiple of the number of consumer hosts is beneficial. >>> Because of this, whenever we add or remove consumer hosts, we have to >>> change the number of partitions in the server config. >>> >>> What are best practices for changing the number of partitions? It seems >>> like adding partitions is fine but removing partitions would result in >> data >>> loss - am I right? Is that avoidable? Is it preferable to bring in new >>> servers with new partitions? Anything else I should keep in mind on this >>> issue? >>> >>> >>> Thanks! >>> >>> David >>
Re: Logo
+1 Sent from my iPhone On Dec 19, 2012, at 3:31 PM, "Sybrandy, Casey" wrote: > If done right, a parody of the Kmart logo might look good as well. > > -Original Message- > From: Neha Narkhede [mailto:neha.narkh...@gmail.com] > Sent: Wednesday, December 19, 2012 5:24 PM > To: users@kafka.apache.org > Subject: Re: Logo > > I really like the black and white sketches with some stylized K on it :-) > http://3.bp.blogspot.com/-XEU25Q9oRlo/TzcM5VVa9YI/AF8/f3O4LLPkHNs/s1600/drawings.jpg > > > On Wed, Dec 19, 2012 at 10:24 AM, Jay Kreps wrote: > >> Yeah it comes from the author. There is no direct connection--i.e. >> Franz Kafka did not write about unifying all your data in a >> high-throughput distributed log thing as far as I know. :-) >> >> We had previous gone with a Harry Potter theme at LinkedIn, but after >> a half dozen projects had kind of exhausted the good Harry Potter >> related names. "Kafka" sounds cool and kind of kept the literary theme. >> >> -Jay >> >> >> On Wed, Dec 19, 2012 at 10:13 AM, David Arthur wrote: >> >>> Somewhat tangential, but where did the name Kafka come from? Is it a >>> reference to Franz Kafka and The Metamorphosis (or other work)? I >>> ask, because knowing how the name "Kafka" relates to a distribute >>> pub/sub >> system >>> might help inspire ideas for a logo. >>> >>> -David >>> >>> >>> On 12/19/12 1:03 PM, Jay Kreps wrote: >>> A few people have asked about getting a logo for Kafka. I think this >> would be a great idea! Oliver has kindly offered to have a someone at Datadog take a shot at >> it. If that doesn't pan out I am happy to do a 99designs contest to get us more options. It would be good for people to use this thread to give ideas or thoughts on what might be good. I will follow up with mine. I will give this a day >> or so and then try to summarize the suggestions/guidance so that any potential designer isn't overwhelmed by contradictory suggestions. -Jay >>
