Re: What is the procedure of upgrading producers from 0.8 to 0.10

Thanks. Got it same rules apply to producer. Thanks again. On Mon, Sep 26, 2016 at 2:29 PM, Alexis Midon < alexis.mi...@airbnb.com.invalid> wrote: > the official recommendations are here > http://kafka.apache.org/documentation.html#upgrade_10 > > > On Fri, Sep 23, 2016 at

What is the procedure of upgrading producers from 0.8 to 0.10

Hello we have a producer that is written in c language to send data to kafka using 0.8 protocol. We now need to upgrade since protocol has changed. We will upgrade broker first to 0.10 version and set log.message.format.version=0.8.1.1. What is the right approach of upgrading producer to avoid dow

Does old consumer api supports 0.10 message format after jar update ?

Hello Everyone. We are in the process of upgrading our brokers and consumers from 0.8.1.1 to 0.10. Our consumer use 0.8 high level or simple consumer api. Will old consumer api support new message format introduced in 0.10 after we upgrade consumer jars to 0.10? Thanks so much in advance

Re: Benchmarking kafka performance

Good afternoon. Any suggestions regarding benchmark tool would be greatly appreciated. Thanks On Mon, Sep 19, 2016 at 8:18 AM, Vadim Keylis wrote: > Good morning. Which benchmarking tools we should use to compare > performance of 0.8 and 0.10 versions? Which metrics should we m

Benchmarking kafka performance

Good morning. Which benchmarking tools we should use to compare performance of 0.8 and 0.10 versions? Which metrics should we monitor ? Thanks in advance, Vadim

Upgrading from 0.8 to 0.10

We are planning to upgrade from 0.8 to 0.10. I am reading about possible performance impact of the upgrade. Do the performance hit due to conversation to 0.8 format or its related to something else? Would someone suggest best way to measure the performance hit? Thanks so much in advance, Vadim

Re: Consumer api 0.10 backwards compatible with Kafka 0.8

Thanks so much to both of you. Vadim On Wed, Sep 7, 2016 at 2:07 AM, Ismael Juma wrote: > Hi Vadim, > > You have to upgrade the brokers first and then the clients. So, you can use > 0.8 clients with 0.10 brokers, but _not_ 0.10 clients with 0.8 brokers. > > On Wed, Sep 7, 2016

Consumer api 0.10 backwards compatible with Kafka 0.8

Hello. We are running kafka server 0.8. We are considering upgrading to the latest version of kafka. Does latest consumer 0.10 backward compatible with older version of Kafka? What is the right approach of upgrading Kafka servers and consumers? Thanks in advance.

Offset Gap

Hello. We are using kafka 0.8.1. I implemented client based on simple consumer. Does the offset always guarantee to be sequential i.e incremented by 1? Under what circumstances would there be a gap i.e differece between current and next offset larger then 1? What is the best way to handle such cas

java.lang.NoClassDefFoundError: Could not initialize class com.yammer.metrics.Metrics

I suddenly started getting this error using the code that was perfectly working a day before. The library containing yammer metrics exists and part of mvn build. Please help how to resolve the issue or disable yammer.metrics. Thanks in advance Exception in thread "Thread-3" java.lang.NoClassDefFo

Kafka 0.8.1.1

We are using kafka 0.8.1.1. I am writing java consumer that will use simple api and keep offset internally. I would like to write automated tests, that will use embedded into java junit kafka and zookeeper to test against. Would you guys point to any working examples. Thanks so much in advance, Va

framework to load streamed data from kafka into relational database

Good evening. Can someone suggest existing framework that allows to reliably load data from kafka into relation database like Oracle in real time? Thanks so much in advance, Vadim

Re: How to build 0.8.1 from source

Thanks so much Neha. On Mon, Mar 24, 2014 at 10:13 AM, Neha Narkhede wrote: > We moved to gradle in 0.8.1. See the README for instructions on gradle > build - https://github.com/apache/kafka/blob/0.8.1/README.md > > Thanks, > Neha > > > On Mon, Mar 24, 2014 at 10:07

How to build 0.8.1 from source

Good morning. I am trying to build 0.8.1 from source because I like to include some jars that are not part of official distribution. I cloned 0.8.1 tag git clone -b 0.8.1.0 https://git-wip-us.apache.org/repos/asf/kafka.git. The issue I have its missing sbt. How can I build from source? Please advis

Re: Best ways to unit test consumer that is based on simpleConsumer

t; actually doesn't do much with the fetch request and response. Take a look > at the fetch() API on SimpleConsumer. It merely returns the response back > to the user. > > Thanks, > Neha > > > On Wed, Mar 12, 2014 at 11:58 PM, Vadim Keylis >wrote: > > > Addi

Re: Best ways to unit test consumer that is based on simpleConsumer

Additional information. We currently using kafka 0.8 and client will be written in java. On Wed, Mar 12, 2014 at 10:41 PM, Vadim Keylis wrote: > Good evening. I am writing consumer that is based on SimpleConsumer. I > would like to write integration/unit tests that simulate different

Best ways to unit test consumer that is based on simpleConsumer

Good evening. I am writing consumer that is based on SimpleConsumer. I would like to write integration/unit tests that simulate different error responses from kafka to make sure my code handles it properly. How do you guys advised to do this within my unit test? Do you have some kind of mock kafka

Re: Fetch Response for invalid offset returns success.

; > Guozhang > > > On Fri, Feb 28, 2014 at 3:18 PM, Vadim Keylis > wrote: > > > Good afternoon. I am writing my own client that is based on simple > > consumer. I implemented method that is based on example in the kafka > > documents that sends request to kafka to find

Re: Fetch Response for invalid offset returns success.

I am using Kafka 0.8 On Fri, Feb 28, 2014 at 4:30 PM, Neha Narkhede wrote: > What version of Kafka are you using? > > > On Fri, Feb 28, 2014 at 3:18 PM, Vadim Keylis > wrote: > > > Good afternoon. I am writing my own client that is based on simple > > consumer

Fetch Response for invalid offset returns success.

Good afternoon. I am writing my own client that is based on simple consumer. I implemented method that is based on example in the kafka documents that sends request to kafka to find an offset. I am trying to find offsets after given offset. The api returns success even those offset does not exist b

Re: How to force producer and high level consumer to use different Ethernet cards on the kafka broker node

? Thanks, Vadim On Wed, Jan 15, 2014 at 9:33 PM, Jun Rao wrote: > Currently, Kafka broker only supports one port for both the produce and the > consumer clients. > > Thanks, > > Jun > > > On Wed, Jan 15, 2014 at 4:15 PM, Vadim Keylis > wrote: > > > Hi. We

How to force producer and high level consumer to use different Ethernet cards on the kafka broker node

Hi. We have kafka setup on server that has two Ethernet cards. Each Ethernet port has its own ip address. We would like to separate traffic and setup producers to communicate through one card and consumers that use high level client through the other card. Would you advise if that configuration pos

Re: Commit Offset per topic

Joel. Just to be clear consumer re-balances you mentioning in the FAQ refers to consumer restart, am I correct? On Wed, Nov 6, 2013 at 5:51 PM, Joel Koshy wrote: > This question seems to come up often - added this to the FAQ. > > > https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Idon%

Re: High Level Consumer commit offset

gt; > > > > > Thanks, > > > Neha > > > > > > > > > On Mon, Nov 4, 2013 at 4:38 PM, Guozhang Wang > > wrote: > > > > > > > And exceptions you saw from the broker end, in server log? > > > > > > > >

Re: High Level Consumer commit offset

gt; > > Guozhang > > > > > > On Mon, Nov 4, 2013 at 4:27 PM, Vadim Keylis > > wrote: > > > > > Thanks for confirming, but that not behavior I observe. My consumer > does > > > not commit data to kafka. It get messages sent to kafka. Once >

Re: High Level Consumer commit offset

gt; On Mon, Nov 4, 2013 at 4:38 PM, Guozhang Wang wrote: > > > And exceptions you saw from the broker end, in server log? > > > > Guozhang > > > > > > On Mon, Nov 4, 2013 at 4:27 PM, Vadim Keylis > > wrote: > > > > > Thanks for confirm

Re: High Level Consumer commit offset

; > Guozhang > > > On Mon, Nov 4, 2013 at 2:42 PM, Vadim Keylis > wrote: > > > Good afternoon. I was under impression if auto commit set to false then > > once consumer is restarted then logs would be replayed from the > beginning. > > Is that correct? > > > > Thanks, > > Vadim > > > > > > -- > -- Guozhang >

High Level Consumer commit offset

Good afternoon. I was under impression if auto commit set to false then once consumer is restarted then logs would be replayed from the beginning. Is that correct? Thanks, Vadim

Monitoring questions.

Good afternoon. I have a few questions about kafka monitoring. 1. In the kafka monitoring section ( http://kafka.apache.org/documentation.html#monitoring) "kafka.server":name="UnderReplicatedPartitions", type="ReplicaManager" sets normal value

Re: Kafka Monitoring

Jun. Greatly appreciated. On Wed, Sep 4, 2013 at 10:12 PM, Jun Rao wrote: > Updated the doc at http://kafka.apache.org/documentation.html#monitoring > > Hopefully that answers your questions. > > Thanks, > > Jun > > > On Tue, Sep 3, 2013 at 11:16 PM, Vadim Keyli

Kafka Monitoring

Good evening. I have read through section of monitoring. I tried to map each section to corresponding JMX attribute. I will appreciate if you answer a few questions bellow. Thanks so much in advance, Vadim What this JMX "kafka.controller":type="KafkaController",name="ActiveControllerCount" fo

Re: Out of memory exception

; > > On Fri, Aug 30, 2013 at 10:39 PM, Vadim Keylis >wrote: > > > I followed linkedin setup example in the docs and located 3g for heap > size. > > > > java -Xmx3G -Xms3G -server -XX:+UseCompressedOops -XX:+UseParNewGC > > -XX:+UseConcMarkSwe

Out of memory exception

I followed linkedin setup example in the docs and located 3g for heap size. java -Xmx3G -Xms3G -server -XX:+UseCompressedOops -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC - After a day of normal run scenario I discover

Re: Instances became unresponsive

How to prevent or proper configure since any broker can become unresponsive? Thanks, Vadim On Tue, Aug 27, 2013 at 11:50 PM, Vadim Keylis wrote: > Hello Jun. Unfortunately I do not have logs from broker 6 to find out > reasons for it to be unresponsive, but yes it was not healthy. I fo

Re: Instances became unresponsive

It seems the replica fetch thread died because of socket timeout (defaults > to 30 secs). Was broker 6 healthy at that point? > > Thanks, > > Jun > > > On Tue, Aug 27, 2013 at 11:36 AM, Vadim Keylis >wrote: > > > We do not use controlled shutdown through JMX, its

Re: Instances became unresponsive

ReplicaFetcherThread-0-6], Shutting down (kafka.server.ReplicaFetcherThread) [2013-08-26 12:38:23,821] INFO Closing socket connection to /10.15.11.24. (kafka.network.Processor) [2013-08-26 12:38:23,899] INFO [ReplicaFetcherThread-0-6], Stopped (kafka.server.ReplicaFetcherThread) [2013-08-26 12:3

Re: Instances became unresponsive

will need to run the > leader balance tool. > > Thanks, > > Jun > > > On Mon, Aug 26, 2013 at 10:12 PM, Vadim Keylis wrote: > >> Somehow I am getting my instances of kafka to crash. I started kafka >> instances one by one and they started successfully

Instances became unresponsive

Somehow I am getting my instances of kafka to crash. I started kafka instances one by one and they started successfully. Later it some how two of 3 instances became completely unresponsive. The process is running, but connnection over jmx or taking heat dump not possible. The last one some what res

Re: No leaders elected after topics created

Sorry Jun. Somehow logs from that particular event got wiped out. On Thu, Aug 22, 2013 at 8:59 PM, Jun Rao wrote: > Any error in the controller and state-change log? > > Thanks, > > Jun > > > On Thu, Aug 22, 2013 at 10:01 AM, Vadim Keylis >wrote: > > >

Re: Producer.send questions

Jun. In general how one will recover from NotLeaderForPartitionException? What are the steps? Thanks, Vadim On Fri, Aug 23, 2013 at 9:07 PM, Jun Rao wrote: > For the most part, only SocketExceptions and NotLeaderForPartitionException > are recoverable. MessageSizeTooLargeException may be recov

Re: delete a topic

; To: users@kafka.apache.org >> Subject: Re: delete a topic >> >> In production, we are not deleting topics yet. In test environments, if we >> have to delete topics, we wipe out the cluster. If this is a feature that >> most users are blocked on, I think it makes sense

Re: How to rebuild missing index?

out is how you reached this > state ? > > Thanks, > Neha > > > On Fri, Aug 23, 2013 at 10:13 AM, Vadim Keylis >wrote: > > > We got to this state somehow. Is there way to rebuild index or tell kafka > > recovery process skip topics that are incomplete > >

Re: Kafka Yammer Metrics

eha > > > On Fri, Aug 23, 2013 at 10:27 AM, Vadim Keylis >wrote: > > > Good morning. i have not used yammer metrics until we deploy kafka. Csv > > output is great, but sending data to graphite or similar straight from > > kafka would be better. We can do our custo

Kafka Yammer Metrics

Good morning. i have not used yammer metrics until we deploy kafka. Csv output is great, but sending data to graphite or similar straight from kafka would be better. We can do our custom implementation that queries jmx and sends data, but I would be curious how community does it. is there module th

How to rebuild missing index?

We got to this state somehow. Is there way to rebuild index or tell kafka recovery process skip topics that are incomplete [2013-08-13 17:20:08,953] INFO [Log Manager on Broker 7] Loading log 'm3_registration-29' (kafka.log.LogManager) [2013-08-13 17:20:08,992] FATAL Fatal error during KafkaServerS

Re: No leaders elected after topics created

gt; > On Thu, Aug 22, 2013 at 11:30 AM, Vadim Keylis >wrote: > > > Hello Neha. The topics should of been available since I ran list-topics > > script 5 hours later after creating topics. > > I am reffering to kafka-delete-topic.sh script that I got from kafka git > > re

Re: delete a topic

d remove any scripts that give the > > impression of deleting a topic, as Jay suggested. I tried looking under > > bin/ but couldn't find any delete topic or partition script. > > > > Can you tell us which script you are using? > > > > Thanks, > > Neha &g

Re: delete a topic

Jay or Neha. What is the process of deleting the topic if the script included in the repository creates false impression? Thanks, Vadim On Thu, Aug 22, 2013 at 11:14 AM, Jay Kreps wrote: > We should really remove the delete script from 0.8 if we plan to release > without delete support. Right

Re: No leaders elected after topics created

e partition script are you referring to? We don't have a delete > topic feature in Kafka yet. So you can only create more partitions, but not > delete them. > > Thanks, > Neha > > > On Thu, Aug 22, 2013 at 10:01 AM, Vadim Keylis >wrote: > > > I ran command to

No leaders elected after topics created

I ran command to create 300 topics sequentially. Kafka create topic script return with topic created successfully message. However running kafka-list-topics.sh script showed to me that no leaders been selected and no servers listed in ISR. First inspection of the logs showed no obvious signs of the

Re: Kafka startup/restart process

/tejasapatil/kafka/blob/0.8.0-beta1-candidate1/core/src/main/scala/kafka/server/KafkaServer.scala#L165 > > > On Mon, Aug 19, 2013 at 10:19 PM, Vadim Keylis >wrote: > > > Neha. Thanks so much for explaining. That leaves only one open question. > > How do you validate tha

Re: Kafka startup/restart process

remotely on Kafka brokers in > production. So we prefer using the controlled.shutdown.enable method. > > Thanks, > Neha > > > On Mon, Aug 19, 2013 at 12:34 PM, Vadim Keylis >wrote: > > > What is preferred method for control shutdown using admin tool or setting > > as

Re: Failed to start preferred replica election

ders to them. You can run > kafka-list-topics with the --under-replicated-count option to print the > list of under replicated partitions. > > Please can you file a bug to improve the error reporting of this tool? > > Thanks, > Neha > > > On Mon, Aug 19, 2013 at 12:26 P

Re: Kafka startup/restart process

What is preferred method for control shutdown using admin tool or setting as flag "controlled.shutdown.enable" to true? What is the advantage of using one verses the other? Thanks, Vadim On Sun, Aug 18, 2013 at 11:05 PM, Vadim Keylis wrote: > thanks so much. Greatly appreciated.

Failed to start preferred replica election

I have a cluster of 3 kafka servers. Replication factor is 3. Two out of 3 servers were shutdown and traffic was sent to only one server that was up. I brought second host up and it says according to logs that server has started. I ran ./kafka-list-topic.sh --zookeeper Still was showing leaders a

Re: Kafka startup/restart process

t; Neha > > > On Sun, Aug 18, 2013 at 8:33 PM, Vadim Keylis > wrote: > > > Neha. Thanks so much for response. How can I get under replicated > partition > > count during control shutdown that is configured in the property file? > > > > Thanks, > >

Re: Kafka startup/restart process

ime > the broker takes to rejoin the cluster. > > Thanks, > Neha > > > On Sun, Aug 18, 2013 at 3:33 PM, Vadim Keylis > wrote: > > > Good afternoon. We are running kafka on centos linux. I enabled > controlled > > shutdown in the property file. We are star

Kafka startup/restart process

Good afternoon. We are running kafka on centos linux. I enabled controlled shutdown in the property file. We are starting/stopping kafka using init script. The init script will issue term signal first followed 3 seconds later by kill signal. Is that right process to shutdown kafka? Which startup/sh

Re: Leader election in the failure mode

How often should we run preferred-replica-election tool? On Sun, Aug 18, 2013 at 3:18 PM, Vadim Keylis wrote: > Thanks so much Jay for clarifying that. > > > On Thu, Aug 15, 2013 at 8:26 PM, Jay Kreps wrote: > >> Leader election is automatic. The preferred-replica-electio

Re: Leader election in the failure mode

of partitions. > > -Jay > > On Thu, Aug 15, 2013 at 6:56 PM, Vadim Keylis > wrote: > > > Hello guys. I am not clear about leader election rules in case of server > > or service failure. If one of the servers went down that was a leader for > > some partitions. Will

Leader election in the failure mode

Hello guys. I am not clear about leader election rules in case of server or service failure. If one of the servers went down that was a leader for some partitions. Will new leader be elected automatically on server that has replica or manual intervention is required like running leader election

Re: Kafka crashed after multiple topics were added

Jay > > > On Thu, Aug 15, 2013 at 3:41 PM, Vadim Keylis > wrote: > > > Jay. Thanks so much for explaining. What is the optimal number of > > partitions per topic? What are the reasoning were behind your guys choice > > of 8 partitions per topic? > > > &

Re: Kafka crashed after multiple topics were added

as * 2 (index file and log file) + #open > sockets > > -Jay > > > On Thu, Aug 15, 2013 at 11:49 AM, Vadim Keylis >wrote: > > > Good Morning Joel. Just to understand clearly how to predict number of > open > > files kept by kafka. > > > > That i

Re: Kafka crashed after multiple topics were added

ention period and number of > simultaneous producers/consumers. > > In your case it seems you have 150 topics, 36 partitions, 3x replication - > with that configuration you will definitely need to up your file handle > limit. > > Thanks, > > Joel > > On Wednesd

Re: new log.dirs property (as opposed to log.dir)

Good morning Jay. When you say delete directory from the list, did you mean from file system? Can I see through JMX which partitions online and which one are not? Thanks On Thu, Aug 15, 2013 at 10:04 AM, Jason Rosenberg wrote: > Thanks Jay, I'll do some testing with this and report back. > > J

Re: How many replicas should be up in order for Kafka to accept message?

sh messages > as long as your ack <=1 (meaning no need to wait for more than one replica > to commit before returning the produce request). > > Guozhang > > On Wed, Aug 14, 2013 at 6:10 PM, Vadim Keylis wrote: > >> We have cluster of 3 Kafka nodes. All topics create

Re: How many replicas should be up in order for Kafka to accept message?

I think we figure out the issue Thanks Sent from my iPhone On Aug 14, 2013, at 6:10 PM, Vadim Keylis wrote: > We have cluster of 3 Kafka nodes. All topics created with replication factor > of 3. Lets say two crashed for what ever reason. I ran tool prefer replica > election leade

How many replicas should be up in order for Kafka to accept message?

We have cluster of 3 Kafka nodes. All topics created with replication factor of 3. Lets say two crashed for what ever reason. I ran tool prefer replica election leaders script. it look to me as leaders were moved to the node that is up. However once trying to publish exception about replicas is

Re: Kafka crashed after multiple topics were added

One more question. What is the optimal number partition per topic to have? On Wed, Aug 14, 2013 at 9:47 AM, Vadim Keylis wrote: > Joel thanks so much. > Do you guys have hard set limit on a maximum topics Kafka can support. Are > there any other OS level settings I should be concerned

Re: Kafka crashed after multiple topics were added

ed to up your file handle > limit. > > Thanks, > > Joel > > On Wednesday, August 14, 2013, Vadim Keylis wrote: > > > Good morning Jun. Correction in terms of open file handler limit. I was > > wrong. I re-ran the command ulimit -Hn and it shows 10240. Which brings &

Re: Kafka crashed after multiple topics were added

Wed, Aug 14, 2013 at 8:19 AM, Vadim Keylis wrote: > Good morning Jun. We are using Kafka 0.8 that I built from trunk in June > or early July. I forgot to mention that running ulimit on the hosts shows > open file handler set to unlimited. What are the ways to recover from last > error

Re: Kafka crashed after multiple topics were added

ts close to the limit. > > Not sure why you get the second error on restart. Are you using the 0.8 > beta1 release? > > Thanks, > > Jun > > > On Tue, Aug 13, 2013 at 11:04 PM, Vadim Keylis wrote: > >> We have 3 node kafka cluster. I initially created 4 topi

Kafka crashed after multiple topics were added

We have 3 node kafka cluster. I initially created 4 topics. I wrote small shell script to create 150 topics. TOPICS=$(< $1) for topic in $TOPICS do echo "/usr/local/kafka/bin/kafka-create-topic.sh --replica 3 --topic $topic --zookeeper $2:2181/kafka --partition 36" /usr/local/kafka/bin/kafka

Zookeeper setup

I am trying to setup kafka service and connect to zookeeper that would be shared with Other projects. Can someone advice how to configure namespace in kafka and zookeeper. Thanks so much Sent from my iPad

Re: LeaderNotAvailableException

; > Thanks, > > Jun > > > On Wed, Aug 7, 2013 at 7:50 AM, Vadim Keylis > wrote: > > > Jun, > > > > What the process now if we want to add and remove servers? How can I > > recover from error in mean time? > > When is the final release

Re: LeaderNotAvailableException

ker to another. It's still being tested and improved. It will > be complete in the 0.8 final release. > > Thanks, > > Jun > > > On Tue, Aug 6, 2013 at 11:18 PM, Vadim Keylis > wrote: > > > Your assumption is correct. However I was under impression that shutdo

Re: LeaderNotAvailableException

lytics_data_log > > > On Tue, Aug 6, 2013 at 10:08 PM, Vadim Keylis wrote: > >> We are using kafka08 beta1. We had 6 kafka in the tier. I have replication >> set to 3. Then 3 servers were removed using control shutdown method. I am >> getting the error bellow afte

LeaderNotAvailableException

We are using kafka08 beta1. We had 6 kafka in the tier. I have replication set to 3. Then 3 servers were removed using control shutdown method. I am getting the error bellow after that. What went wrong during shutdown? How to recover from the error? What steps to take in order to avoid in the futu

Moving raw logs from kafka into hadoop using camus

Good afternoon. I am new to camus and would like to use it to move raw data from kafka to hadoop. Do I have to pre-create avro schema in advance or it automatically created for me? What is the role of the decoder class specified by camus.message.decoder.class in property file? Do I need to impl

Re: Does ShutdownBroker returns meaningful return code

and.) > > On Wed, Jul 10, 2013 at 4:06 PM, Vadim Keylis > wrote: > > We have deployed kafka 0.8 beta1. It was my understanding that > > ShutdownBroker program needs be used to initiate proper shutdown of the > > server. We are going to use this script in automated fashio

Does ShutdownBroker returns meaningful return code

We have deployed kafka 0.8 beta1. It was my understanding that ShutdownBroker program needs be used to initiate proper shutdown of the server. We are going to use this script in automated fashion. Does the script return meaningful error code that can be capture by calling script and act up on? What

Re: Kafka Important Metrics

> > can directly feed into your monitoring system. E.g., if you use > > graphite, you can write a KafkaGraphiteReporter that wraps the > > GraphiteReporter that coda hale metrics provides. > > > > Joel > > > > On Mon, Jul 1, 2013 at 11:21 AM, Vadim

Re: Kafka/Hadoop consumers and producers

Jay, What is the difference between this project and Camus? Which advantages to use for loading log entries from kafka into Hadoop ? Vadim Sent from my iPhone On Jul 2, 2013, at 5:01 PM, Jay Kreps wrote: > We currently have a contrib package for consuming and producing messages > from mapredu

Re: Log4J setting for kafka

Thanks On Mon, Jul 1, 2013 at 11:44 AM, David DeMaagd wrote: > The danger of using a size based rollover (unless you set the size and > log rollover to be fairly high) is that in case of problems, the actual > cause of the problem might get rolled off the end by the time you get to > it (kafka c

Log4J setting for kafka

Good morning. The log4j property file included wit distribution contains daily log rotation. Are there any reason you guys chose daily rotation vs RollingFileAppender? Thanks, Vadim

.IOException: Unable to create /home/kafka/metrics/NumDelayedRequests.csv

I am getting this exception even those permission properly set and empty file is created. Other metric files created without problems. java.io.IOException: Unable to create /home/kafka/metrics/NumDelayedRequests.csv at com.yammer.metrics.reporting.CsvReporter.createStreamForMetric(CsvRepor

Kafka Important Metrics

Good morning. What is the best way to monitor kafka through jmx or by enabling kafka.csv.metrics.reporter.enabled? What are the important metrics in JMX to watch for and/or graph? What are the important metrics in csv files to watch for and/or graph? Thanks, Vadim

Re: Fetch request with correlation id 1171437 from client ReplicaFetcherThread-0-1 on partition [meetme,0] failed due to Leader not local for partition

Thanks, > > Jun > > > On Fri, Jun 28, 2013 at 10:57 AM, Vadim Keylis >wrote: > > > Good morning. I have a cluster of 3 kafka nodes. They were both running > at > > the time. I need it to make configuration change in the property file and > > restart kafk

Re: Fetch request with correlation id 1171437 from client ReplicaFetcherThread-0-1 on partition [meetme,0] failed due to Leader not local for partition

> > up > > > > the new leader information. If should settle shortly. > > > > > > > > -- > > > > Dave DeMaagd > > > > ddema...@linkedin.com | 818 262 7958 > > > > > > > > (vkeylis2...@gmail.com - Fri, Jun

Re: Fetch request with correlation id 1171437 from client ReplicaFetcherThread-0-1 on partition [meetme,0] failed due to Leader not local for partition

8 262 7958 > > (vkeylis2...@gmail.com - Fri, Jun 28, 2013 at 11:08:46AM -0700) > > I want to clarify that I restarted only one kafka node, all others were > > running and did not require restart > > > > > > On Fri, Jun 28, 2013 at 10:57 AM, Vadim Keylis >wrote:

Re: Fetch request with correlation id 1171437 from client ReplicaFetcherThread-0-1 on partition [meetme,0] failed due to Leader not local for partition

I want to clarify that I restarted only one kafka node, all others were running and did not require restart On Fri, Jun 28, 2013 at 10:57 AM, Vadim Keylis wrote: > Good morning. I have a cluster of 3 kafka nodes. They were both running at > the time. I need it to make configuration cha

Fetch request with correlation id 1171437 from client ReplicaFetcherThread-0-1 on partition [meetme,0] failed due to Leader not local for partition

Good morning. I have a cluster of 3 kafka nodes. They were both running at the time. I need it to make configuration change in the property file and restart kafka. I have not broker shutdown tool, but simple used pkill -TERM -u ${KAFKA_USER} -f kafka.Kafka. That suddenly cause the exception. How t

Re: Kafka 0.8 Failover Behavior

Jun, Does kafka provides ability to configure broker to be in in-sync before become availalble? Is it possible in case of all brokers crash to find out which node has the most recent data to initiate proper startup procedure? Thanks, Vadim On Fri, Jun 21, 2013 at 8:24 PM, Jun Rao wrote: > Hi,

Re: Truncated messages in fetch response

Jay. I assume this is problem exists in the consumer. How this can this problem be triggered so I could test my high level consumer. Thanks On Jun 26, 2013, at 9:21 AM, Jay Kreps wrote: > Yeah, that is true. I thought I documented that, but looking at the > protocol docs, it looks like I didn'

Re: About Kafka Users Group around Hadoop Summit

Good morning Jun. I am interested as well. Please post details how to signup for this meetup. On Thu, Jun 20, 2013 at 7:59 AM, Jun Rao wrote: > Because of the limited venue in the conference hotel, we decided to host > our user group meeting at LinkedIn in MountainView Thursday (Jun. 27) > even

Re: Code Example for createMessageStreams

you can find the example in the following wiki. > > https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example > > Thanks, > > Jun > > > On Tue, Jun 4, 2013 at 12:56 AM, Vadim Keylis wrote: > >> Good evening. Would it be possible to get samp

Re: Compression in kafka 0.8

On Tue, Jun 4, 2013 at 9:55 PM, Vadim Keylis > wrote: > > > Good evening. I would like to preserve logs in kafka for 2 days. Due to > the > > size of data it must be compressed otherwise it will take enormous amount > > of disk space. Its my understanding that the only

Compression in kafka 0.8

Good evening. I would like to preserve logs in kafka for 2 days. Due to the size of data it must be compressed otherwise it will take enormous amount of disk space. Its my understanding that the only way to store messages in kafka compressed is to enable compression on the producer side, is that co

Code Example for createMessageStreams

Good evening. Would it be possible to get sample code where the api bellow is used in the consumer with sample of Decoder. public Map>> createMessageStreams(Map topicCountMap, Decoder keyDecoder, Decoder valueDecoder); Thanks so much in advance, Vadim

Send multiple messages to multiple topics in one request

Version 7 of Kafka allowed to send messages to multiple topics within one request. I assume I can do samething in version 8, am I correct? Does my code piece look correct? List> topicsMsgsList = new ArrayList(); KeyedMessage data1 = new KeyedMessage(topic1, msg); KeyedMessage data2 = new KeyedMessa

  1   2   >