Re: Producer not distributing across all partitions

>Swapnil > >What do you mean by "I did a local test today that showed that choosing >DefaultPartitioner with >null key in the messages appended data to multiple partitions"? > >Are messages being duplicated across partitions? > >-Chetan > > >On Sat, Sep

Re: Producer not distributing across all partitions

Hi Joe, Drew, In 0.8 HEAD, if the key is null, the DefaultEventHandler randomly chooses an available partition and never calls the partitioner.partition(key, numPartitions) method. This is done in lines 204 to 212 of the github commit Drew pointed to, though that piece of code is slightly differen

Re: Producer not distributing across all partitions

Hi Prashant, I tried a local test using a very short topic.metadata.refresh.interval.ms on the producer. The server had two partitions and both of them appended data. Could you check if you have set the topic.metadata.refresh.interval.ms on your producer to a very high value? Swapnil On 9/13/13

Re: Mirror maker doesn't replicate new topics

Hi Rajasekar, You said that ConsumerOffsetChecker shows that new topics are successfully consumed and the lag is 0. If that's the case, can you verify that there is data on the source cluster for these new topics? If there is no data at the source, MirrorMaker will only assign consumer streams to

Re: Kafka Web Console

Hi Samir, https://issues.apache.org/jira/browse/KAFKA-266 has been filed for this purpose. We haven't made substantial progress yet, but seems like Guy was working on it. Would you mind putting a comment on the JIRA? Thanks, Swapnil On 4/30/13 12:18 AM, "Samir Madhavan" wrote: >Hi, > >Wanted

We have renamed certain Zookeeper and Producer configs

Hello everyone, If you update your Kafka 0.8 to HEAD, please note that we have renamed the following configs (https://issues.apache.org/jira/browse/KAFKA-871): On the broker and consumer: * zk.connect has been renamed to zookeeper.connect * zk.session.timeout.ms has been renamed to zook

Re: Kafka broker not respecting log.roll.hours?

@Dan: Upon restart of the broker, if a segment already has data, the broker resets the firstAppendTime of the segment to the time when that segment's file handles are being loaded into memory. Thus as you correctly explained, every time you shut down a broker, the broker essentially forgets the fir

Re: kafka.javaapi.producer.SyncProducer Confusion

Hi Samir, In 0.8, we don't have SyncProducer as a public api. You can use kafka.javaapi.producer.Producer to send the data synchronously or asynchronously. The quick start page is outdated, and we plan to update a few pages soon (before the 0.8 BETA release). In fact, we would love to have contr

Re: Message size issue

Hi Navneet, Your payload size is larger than the maximum allowed size of a message on the Producer/Broker, 100 is the current default value. Currently the check for max allowed message size is present on both the producer and the broker. So you can override the property "max.message.size" in P

Re: producer creating multiple log files after message sending

Otherwise if you are using the high level consumer (ZookeeperConsumerConnector) in code, you will have to start a new consumer with a different group id. Thanks, Swapnil On 4/18/13 2:09 AM, "Swapnil Ghike" wrote: >Snehalata, > >Are you using the console consumer? Yo

Re: producer creating multiple log files after message sending

Snehalata, Are you using the console consumer? You can pass a --from-beginning flag to the console consumer to get all the messages posted to that topic. Thanks, Swapnil On 4/18/13 1:32 AM, "Snehalata Nagaje" wrote: >Hi Neha, > >I was able to figure this out, this is happening due to time base

Re: simple producer (kafka 0.7.2) exception

Was a kafka broker running when your producer got this exception? Thanks, Swapnil On 4/7/13 3:15 AM, "Oleg Ruchovets" wrote: >try to execute kafka 0.7.2 and got such exception: > > >log4j:WARN No appenders could be found for logger >(org.I0Itec.zkclient.ZkConnection). >log4j:WARN Please initial

Re: How to view Topic registration info and Partition state info from zookeeper in 0.8?

Hi Jason You can download zookeeper, and use a client as zookeeper/bin/zkCli.sh -server zkHost:zkPort. Then you can use the ls and get functions to list the structure hierarchy or to see the contents at each of the nodes in the hierarchy respectively. The data structures described in the wiki can

Please clean your ZK and Kafka data if you check out 0.8 HEAD

Hi all, Today we checked in a patch that standardizes Zookeeper data to use JSON format (KAFKA-755). If you plan to rebase or clone to use the HEAD of 0.8 branch, your next build will have changes to the way Zk data is read and written. This data

Re: 0.8 build problems

Hi Derek, We probably have a patch for adding assembly at https://issues.apache.org/jira/browse/KAFKA-733. Can you review it? Thanks, Swapnil On 2/21/13 2:46 PM, "Derek Chen-Becker" wrote: >The two simplest approaches (short of parsing SBT output for classpaths) >would be to either use https:/

Re: log file flush?

Correction - The flush happens based on *number of messages* and time limits, whichever is hit first. On 2/19/13 3:50 AM, "Swapnil Ghike" wrote: >The flush happens based on size and time limits, >whichever is hit first.

Re: log file flush?

The messages for a topic are kept in the kafka broker's memory before they are flushed to the disk. The flush happens based on size and time limits, whichever is hit first. If you kill the kafka server process before any message has been flushed to the disk, those messages will be lost. The config