I double posted by accident, sorry. Have another thread discussing this. Thanks! On Dec 22, 2014 11:21 AM, "Jun Rao" <j...@confluent.io> wrote:
> Yes, that's a potential issue. Perhaps we just need to have a lower default > value for metadata.fetch.timeout.ms ? > > Thanks, > > Jun > > On Wed, Dec 17, 2014 at 11:10 PM, Paul Pearcy <paul.pea...@blackboard.com> > wrote: > > > Heya, > > Playing around with the 0.8.2-beta producer client. One of my test > cases > > is to ensure producers can deal with Kafka being down when the producer > is > > created. My tests failed miserably because of the default blocking in the > > producer with regard to metadata.fetch.timeout.ms. The first line of new > > producer is waitOnMetadata which is blocking. > > > > I can handle this case by loading topic meta on init and setting the > > timeout value to very low metadata.fetch.timeout.ms and either throwing > > away messages or creating my own internal queue to buffer. > > > > I'm surprised the metasync isn't done async. If it fails, return that in > > the future/callback. This way the API could actually be considered safely > > async and the producer buffer could try to hold on to things until > > block.on.buffer.full kicks in. You'd probably need a partition callback > > since numPartitions wouldn't be available. > > > > The implication is that people's apps will work fine if first messages > are > > sent while kafka server is up, however, if kafka is down and they restart > > their app, the new producer will block all sends and blow things up if > you > > haven't written your app to be aware of this edge case. > > > > Thanks, > > Paul > > This email and any attachments may contain confidential and proprietary > > information of Blackboard that is for the sole use of the intended > > recipient. If you are not the intended recipient, disclosure, copying, > > re-distribution or other use of any of this information is strictly > > prohibited. Please immediately notify the sender and delete this > > transmission if you received this email in error. > > >
