The difference btw 0.7 and 0.8 is that in 0.7, a producer can send messages to each broker through a vip. In 0.8, the producer has to be able to talk to individual broker directly.
Thanks, Jun On Fri, Jan 11, 2013 at 11:27 AM, Jason Huang <[email protected]> wrote: > Just curious - is there any way to by-pass the cluster metadata and > only use a static broker (or a static broker list)? I thought before > 0.8 we have a way to skip zookeeper and use static broker, right? > > The issue is that I can't start my kafka server with the 208.39.***.** > ip because that's not a real IP. It's a gateway that will direct the > client process to different internal IPs with different port number. > So for 208.39.***.**:9092, we will direct it to internal IP A. And for > another port, we will direct it to another IP B. I am not sure how to > get pass this. > > thanks, > > Jason > > On Fri, Jan 11, 2013 at 11:42 AM, Jun Rao <[email protected]> wrote: > > The producer used 208.39.***.**:9092 to get the metadata, which was > > successful. The metadata indicates that the producer should send messages > > to the broker at address kafka.icare.com:9092, which was unsuccessful. > The > > problem may be item #2 listed in http://kafka.apache.org/faq.html > > > > Thanks, > > > > Jun > > > > On Fri, Jan 11, 2013 at 8:15 AM, Jason Huang <[email protected]> > wrote: > > > >> Hello, > >> > >> I am testing the producer to send a message from my local PC to a > >> remote Kafka server with 0.8.0 installed. However, I was only able to > >> create the log file with that specific topic but not able to write any > >> message to the log file. > >> > >> Here is what the log from the kafka server: > >> > >> [2013-01-11 10:43:59,787] INFO [KafkaApi-1] Auto creation of topic > >> topic_227 with 1 partitions and replication factor 1 is successful! > >> (kafka.server.KafkaApis) > >> [2013-01-11 10:43:59,789] INFO [TopicChangeListener on Controller 1]: > >> New topics: [Set(topic_227)], deleted topics: [Set()], new partition > >> replica assignment [Map([topic_227,0] -> List(1))] > >> (kafka.controller.PartitionStateMachine$TopicChangeListener) > >> [2013-01-11 10:43:59,789] INFO [Controller 1]: New topic creation > >> callback for [topic_227,0] (kafka.controller.KafkaController) > >> [2013-01-11 10:43:59,790] INFO [Controller 1]: New partition creation > >> callback for [topic_227,0] (kafka.controller.KafkaController) > >> ....... > >> [2013-01-11 10:43:59,830] INFO [Log Manager on Broker 1] Created log > >> for topic topic_227 partition 0 in /opt/kafka-0.8.0/data/kafka-logs. > >> (kafka.log.LogManager) > >> [2013-01-11 10:43:59,831] WARN No previously checkpointed > >> highwatermark value found for topic topic_227 partition 0. Returning 0 > >> as the highwatermark (kafka.server.HighwaterMarkCheckpoint) > >> [2013-01-11 10:43:59,831] INFO Replica Manager on Broker 1: Completed > >> the leader state transition for topic topic_227 partition 0 > >> (kafka.server.ReplicaManager) > >> [2013-01-11 10:43:59,831] INFO Replica Manager on Broker 1: Completed > >> leader and isr request LeaderAndIsrRequest(0,0,,1000,Map((topic_227,0) > >> -> PartitionStateInfo(LeaderIsrAndControllerEpoch({ > >> "ISR":"1","leader":"1","leaderEpoch":"0" > >> },6),1)),Set(id:1,host:kafka.icare.com,port:9092),6) > >> (kafka.server.ReplicaManager) > >> [2013-01-11 10:43:59,959] INFO Closing socket connection to > >> /173.162.***.** (kafka.network.Processor) > >> > >> So at this point there is a folder created on the server in the /logs > >> directory named after the topic and partition: /logs/topic_227-0 > >> And there are two files inside: > >> -rw-r--r--. 1 root root 10485760 Jan 11 10:43 00000000000000000000.index > >> -rw-r--r--. 1 root root 0 Jan 11 10:43 00000000000000000000.log > >> > >> > >> However, the content of the actual message was not written to the .log > >> file. I couldn't find any error in the kafka server log. However, from > >> my JBoss log I see the following error: > >> > >> 10:47:11,554 INFO [kafka.client.ClientUtils$] > >> (http-localhost-127.0.0.1-8080-2) Fetching metadata for topic > >> Set(topic_227) > >> 10:47:11,568 INFO [kafka.producer.SyncProducer] > >> (http-localhost-127.0.0.1-8080-2) Connected to 208.39.***.**:9092 for > >> producing > >> 10:47:11,591 INFO [kafka.producer.SyncProducer] > >> (http-localhost-127.0.0.1-8080-2) Disconnecting from > >> 208.39.***.**:9092 > >> 10:47:11,592 WARN [kafka.producer.async.DefaultEventHandler] > >> (http-localhost-127.0.0.1-8080-2) Failed to send to broker -1 with > >> data Map([topic_227,0] -> > >> ByteBufferMessageSet(MessageAndOffset(Message(magic = 0, attributes = > >> 0, crc = 818361588, key = java.nio.HeapByteBuffer[pos=0 lim=1 cap=51], > >> payload = java.nio.HeapByteBuffer[pos=0 lim=46 cap=46]),0))) > >> 10:47:11,695 INFO [kafka.client.ClientUtils$] > >> (http-localhost-127.0.0.1-8080-2) Fetching metadata for topic > >> Set(topic_227) > >> 10:47:11,710 INFO [kafka.producer.SyncProducer] > >> (http-localhost-127.0.0.1-8080-2) Connected to 208.39.***.**:9092 for > >> producing > >> 10:47:11,742 INFO [kafka.producer.SyncProducer] > >> (http-localhost-127.0.0.1-8080-2) Disconnecting from > >> 208.39.***.**:9092 > >> 10:47:11,885 ERROR [kafka.producer.SyncProducer] > >> (http-localhost-127.0.0.1-8080-2) Producer connection to > >> kafka.icare.com:9092 unsuccessful: > >> java.nio.channels.UnresolvedAddressException > >> at sun.nio.ch.Net.checkAddress(Net.java:30) > [classes.jar:1.6.0_33] > >> at > sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:512) > >> [classes.jar:1.6.0_33] > >> at > kafka.network.BlockingChannel.connect(BlockingChannel.scala:57) > >> at kafka.producer.SyncProducer.connect(SyncProducer.scala:139) > >> at > >> kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:154) > >> at > >> > kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:68) > >> > >> > >> The server ip is 208.39.***.** and the server name is kafka.icare.com. > >> I am not sure why it first connected to the server with the ip address > >> and successfully created the topic with default partition. Then it > >> tried to use the server name to connect and thus failed? I feel like I > >> might be missing some setup in config files but I am not sure exactly > >> what I should change. Can anyone point out why this error occurs and > >> why kafka first tried to connect to ip and then server name? > >> > >> thanks! > >> > >> Jason > >> >
