So what do I do? Kill my production servers and rebuild? Restarting all services does nit work. This seems kinda extreme. At this point I feel I have to kill all servers and rebuild.
Thanks On Fri, Dec 18, 2015 at 2:28 AM, Dana Powers <dana.pow...@gmail.com> wrote: > Hi Ben and Marko -- great suggestions re: connection failures and docker. > > The specific error here is: LeaderNotAvailableError: > TopicMetadata(topic='topic-test-production', error=5, partitions=[]) > > That is an error code (5) returned from a MetadataRequest. In this context > it means that the topic did not exist and so the request triggered an > auto-create initialization (i.e., the connection was fine). Topic > initialization tends to take a few seconds to complete, but only needs to > happen once per topic. A retry here is generally fine. This retry should > probably be handled under the covers by the client code. So in this case I > would treat it as a simple kafka-python issue (#488). > > -Dana > > On Thu, Dec 17, 2015 at 4:58 AM, Ben Davison <ben.davi...@7digital.com> > wrote: > > > I probably should of mentioned that this was using Amazon ECS. > > > > On Thu, Dec 17, 2015 at 12:18 PM, Marko Bonaći < > marko.bon...@sematext.com> > > wrote: > > > > > It doesn't have to be FQDN. > > > > > > Here's how I run Kafka in a container: > > > docker run --name st-kafka -p 2181:2181 -p 9092:9092 -e > > > ADVERTISED_HOST=`docker-machine ip dev-st` -e ADVERTISED_PORT=9092 -d > > > spotify/kafka > > > > > > And then you have access to Kafka on the docker host VM from any other > > > machine. > > > BTW I use Spotify's image since it contains both ZK and Kafka, but I > > think > > > the latest version they built is 0.8.2.1, so you might have to build > the > > > new image yourself if you need 0.9, but that's trivial to do. > > > > > > Marko Bonaći > > > Monitoring | Alerting | Anomaly Detection | Centralized Log Management > > > Solr & Elasticsearch Support > > > Sematext <http://sematext.com/> | Contact > > > <http://sematext.com/about/contact.html> > > > > > > On Thu, Dec 17, 2015 at 11:33 AM, Ben Davison < > ben.davi...@7digital.com> > > > wrote: > > > > > > > Hi David, > > > > > > > > Are you running in docker? Are you trying to connect from to a remote > > > box? > > > > We found we could connect locally but couldn't connect from another > > > remote > > > > host. > > > > > > > > (I've just started using kafka also) > > > > > > > > We had the same issue and found out: host.name=<%=@ipaddress%> > needed > > to > > > > be > > > > the FQDN of the box. > > > > > > > > Thanks, > > > > > > > > Ben > > > > > > > > On Thu, Dec 17, 2015 at 5:40 AM, David Montgomery < > > > > davidmontgom...@gmail.com > > > > > wrote: > > > > > > > > > Hi, > > > > > > > > > > I am very concerned about using kafka in production given the below > > > > > errors: > > > > > > > > > > Now issues with myt zookeeper. Other services use ZK. Only kafka > > > fails. > > > > > I have 2 kafka servers using 8.x. How do I resolve? I tried > > > restarting > > > > > services for kafka. Below is my kafka server.properties file > > > > > > > > > > 'Traceback (most recent call last): > > > > > File > > > > > > > > > > > > > > > > > > > > "/usr/local/lib/python2.7/dist-packages/gevent-1.1b6-py2.7-linux-x86_64.egg/gevent/greenlet.py", > > > > > line 523, in run > > > > > result = self._run(*self.args, **self.kwargs) > > > > > File "/var/feed-server/ad-server/pixel-server.py", line 145, in > > > > > send_kafka_message > > > > > res = producer.send_messages(topic, message) > > > > > File "build/bdist.linux-x86_64/egg/kafka/producer/simple.py", > line > > > 52, > > > > in > > > > > send_messages > > > > > partition = self._next_partition(topic) > > > > > File "build/bdist.linux-x86_64/egg/kafka/producer/simple.py", > line > > > 36, > > > > in > > > > > _next_partition > > > > > self.client.load_metadata_for_topics(topic) > > > > > File "build/bdist.linux-x86_64/egg/kafka/client.py", line 383, in > > > > > load_metadata_for_topics > > > > > kafka.common.check_error(topic_metadata) > > > > > File "build/bdist.linux-x86_64/egg/kafka/common.py", line 233, in > > > > > check_error > > > > > raise error_class(response) > > > > > LeaderNotAvailableError: > TopicMetadata(topic='topic-test-production', > > > > > error=5, partitions=[]) > > > > > <Greenlet at 0x7f7acd1654b0: > > > send_kafka_message('topic-test-production', > > > > > '{"adfadfadf)> failed with LeaderNotAvailableError > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > # limitations under the License. > > > > > # see kafka.server.KafkaConfig for additional details and defaults > > > > > > > > > > ############################# Server Basics > > > ############################# > > > > > > > > > > # The id of the broker. This must be set to a unique integer for > each > > > > > broker. > > > > > broker.id=<%=@broker_id%> > > > > > advertised.host.name=<%=@ipaddress%> > > > > > advertised.port=9092 > > > > > ############################# Socket Server Settings > > > > > ############################# > > > > > > > > > > # The port the socket server listens on > > > > > port=9092 > > > > > > > > > > # Hostname the broker will bind to and advertise to producers and > > > > > consumers. > > > > > # If not set, the server will bind to all interfaces and advertise > > the > > > > > value returned from > > > > > # from java.net.InetAddress.getCanonicalHostName(). > > > > > host.name=<%=@ipaddress%> > > > > > > > > > > # The number of threads handling network requests > > > > > num.network.threads=2 > > > > > > > > > > # The number of threads doing disk I/O > > > > > num.io.threads=2 > > > > > > > > > > # The send buffer (SO_SNDBUF) used by the socket server > > > > > socket.send.buffer.bytes=1048576 > > > > > > > > > > # The receive buffer (SO_RCVBUF) used by the socket server > > > > > socket.receive.buffer.bytes=1048576 > > > > > > > > > > # The maximum size of a request that the socket server will accept > > > > > (protection against OOM) > > > > > socket.request.max.bytes=104857600 > > > > > > > > > > > > > > > ############################# Log Basics > > ############################# > > > > > > > > > > # A comma seperated list of directories under which to store log > > files > > > > > log.dirs=/tmp/kafka-logs > > > > > > > > > > # The number of logical partitions per topic per server. More > > > partitions > > > > > allow greater parallelism > > > > > # for consumption, but also mean more files. > > > > > num.partitions=2 > > > > > > > > > > ############################# Log Flush Policy > > > > > ############################# > > > > > > > > > > # The following configurations control the flush of data to disk. > > This > > > is > > > > > among the most > > > > > # important performance knob in kafka. > > > > > # There are a few important trade-offs here: > > > > > # 1. Durability: Unflushed data may be lost if you are not using > > > > > replication. > > > > > # 2. Latency: Very large flush intervals may lead to latency > > spikes > > > > when > > > > > the flush does occur as there will be a lot of data to flush. > > > > > # 3. Throughput: The flush is generally the most expensive > > > operation, > > > > > and a small flush interval may lead to exceessive seeks. > > > > > # The settings below allow one to configure the flush policy to > flush > > > > data > > > > > after a period of time or > > > > > # every N messages (or both). This can be done globally and > > overridden > > > > on a > > > > > per-topic basis. > > > > > > > > > > # The number of messages to accept before forcing a flush of data > to > > > disk > > > > > log.flush.interval.messages=10000 > > > > > > > > > > # The maximum amount of time a message can sit in a log before we > > > force a > > > > > flush > > > > > log.flush.interval.ms=1000 > > > > > > > > > > # Per-topic overrides for log.flush.interval.ms > > > > > #log.flush.intervals.ms.per.topic=topic1:1000, topic2:3000 > > > > > > > > > > ############################# Log Retention Policy > > > > > ############################# > > > > > > > > > > # The following configurations control the disposal of log > segments. > > > The > > > > > policy can > > > > > # be set to delete segments after a period of time, or after a > given > > > size > > > > > has accumulated. > > > > > # A segment will be deleted whenever *either* of these criteria are > > > met. > > > > > Deletion always happens > > > > > # from the end of the log. > > > > > > > > > > # The minimum age of a log file to be eligible for deletion > > > > > log.retention.hours=168 > > > > > > > > > > # A size-based retention policy for logs. Segments are pruned from > > the > > > > log > > > > > as long as the remaining > > > > > # segments don't drop below log.retention.bytes. > > > > > #log.retention.bytes=1073741824 > > > > > > > > > > # The maximum size of a log segment file. When this size is > reached a > > > new > > > > > log segment will be created. > > > > > log.segment.bytes=536870912 > > > > > > > > > > # The interval at which log segments are checked to see if they can > > be > > > > > deleted according > > > > > # to the retention policies > > > > > log.cleanup.interval.mins=1 > > > > > > > > > > ############################# Zookeeper > ############################# > > > > > > > > > > # Zookeeper connection string (see zookeeper docs for details). > > > > > # This is a comma separated host:port pairs, each corresponding to > a > > zk > > > > > # server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002". > > > > > # You can also append an optional chroot string to the urls to > > specify > > > > the > > > > > # root directory for all kafka znodes. > > > > > #zookeeper.connect=localhost:2181 > > > > > zookeeper.connect=<%=@zookeeper%> > > > > > > > > > > > > > > > # Timeout in ms for connecting to zookeeper > > > > > zookeeper.connection.timeout.ms=1000000 > > > > > > > > > > num.replica.fetchers=4 > > > > > default.replication.factor=2 > > > > > delete.topic.enable=true > > > > > unclean.leader.election.enable=true > > > > > > > > > > > > > -- > > > > > > > > > > > > This email, including attachments, is private and confidential. If > you > > > have > > > > received this email in error please notify the sender and delete it > > from > > > > your system. Emails are not secure and may contain viruses. No > > liability > > > > can be accepted for viruses that might be transferred by this email > or > > > any > > > > attachment. Any unauthorised copying of this message or unauthorised > > > > distribution and publication of the information contained herein are > > > > prohibited. > > > > > > > > 7digital Limited. Registered office: 69 Wilson Street, London EC2A > 2BB. > > > > Registered in England and Wales. Registered No. 04843573. > > > > > > > > > > > -- > > > > > > This email, including attachments, is private and confidential. If you > have > > received this email in error please notify the sender and delete it from > > your system. Emails are not secure and may contain viruses. No liability > > can be accepted for viruses that might be transferred by this email or > any > > attachment. Any unauthorised copying of this message or unauthorised > > distribution and publication of the information contained herein are > > prohibited. > > > > 7digital Limited. Registered office: 69 Wilson Street, London EC2A 2BB. > > Registered in England and Wales. Registered No. 04843573. > > >
